Collaborative location/orientation estimation

ABSTRACT

It is inter alia disclosed to estimate at least one of a location and an orientation of a first-type device ( 11 ) at least based on respective location estimates for one or more signal sources ( 10 ), wherein at least one location estimate for a signal source ( 10 ) of the one or more signal sources is determined at least based on respective signals from the signal source ( 10 ) received at one or more second-type devices ( 12 ) and respective locations of the one or more second-type devices ( 12 ), and respective position relationships between the one or more signal sources ( 10 ) and the first-type device ( 11 ).

RELATED APPLICATION

This application was originally filed as PCT Application No.PCT/IB2010/050095 filed Jan. 12, 2010.

FIELD

This invention relates to estimating a location and/or an orientation ofa first-type device collaboratively with the help of at least onesecond-type device.

BACKGROUND

Location and orientation information of a device (for instance a mobiledevice) is important for location based service provision (e.g.navigation and information services) as well as for serviceenhancements.

As an example, accurate knowledge of the location and orientation of adevice is required if the device is connected in an ad-hoc network formulti-view audiovisual content capture. In such a multi-view contentcapture operation, the device may for instance function as a camera,video camera or just as an audio recorder. The device may for instancebe connected to the free viewpoint service provision as a contentcapture client. The knowledge about the viewpoint to the audiovisualsource and the synchronisation of the captured content in multi-devicesystem is important to efficiently store, analyse, process and representthe media.

As a further example, augmented reality services require accuratelocation and orientation of the user of a device to enable meaningfulrendering of the media items within the given location.

SUMMARY OF SOME EMBODIMENTS OF THE INVENTION

In a first aspect of the present invention, a method is disclosed,comprising estimating at least one of a location and an orientation of afirst-type device at least based on

-   -   respective location estimates for one or more signal sources,        wherein at least one location estimate for a signal source of        the one or more signal sources is determined at least based on        respective signals from the signal source received at one or        more second-type devices and respective locations of the one or        more second-type devices, and    -   respective position relationships between the one or more signal        sources and the first-type device.

In this first aspect of the present invention, furthermore a computerprogram is disclosed, comprising program code for performing the methodaccording to the first aspect of the present invention when the computerprogram is executed on a processor. The computer program may forinstance be distributable via a network, such as for instance theInternet. The computer program may for instance be storable or encodablein a computer-readable medium. The computer program may for instance atleast partially represent software and/or firmware of the processor.

In this first aspect of the present invention, furthermore acomputer-readable medium is disclosed, having a computer programaccording to the first aspect of the present invention stored thereon.The computer-readable medium may for instance be embodied as anelectric, magnetic, electro-magnetic, optic or other storage medium, andmay either be a removable medium or a medium that is fixedly installedin an apparatus or device. Non-limiting examples of such acomputer-readable medium are a Random-Access Memory (RAM) or a Read-OnlyMemory (ROM). The computer-readable medium may for instance be atangible medium, for instance a tangible storage medium. Acomputer-readable medium is understood to be readable by a computer,such as for instance a processor.

In this first aspect of the present invention, furthermore an apparatusis disclosed, configured to perform the method according to the firstaspect of the present invention.

In this first aspect of the present invention, furthermore an apparatusis disclosed, comprising means for estimating at least one of a locationand an orientation of a first-type device at least based on

-   -   respective location estimates for one or more signal sources,        wherein at least one location estimate for a signal source of        the one or more signal sources is determined at least based on        respective signals from the signal source received at one or        more second-type devices and respective locations of the one or        more second-type devices, and    -   respective position relationships between the one or more signal        sources and the first-type device.

In this first aspect of the present invention, furthermore an apparatusis disclosed, comprising at least one processor; and at least one memoryincluding computer program code, the at least one memory and thecomputer program code configured to, with the at least one processor,cause the apparatus at least to estimate at least one of a location andan orientation of a first-type device at least based on

-   -   respective location estimates for one or more signal sources,        wherein at least one location estimate for a signal source of        the one or more signal sources is determined at least based on        respective signals from the signal source received at one or        more second-type devices and respective locations of the one or        more second-type devices, and    -   respective position relationships between the one or more signal        sources and the first-type device.

The computer program code included in the memory may for instance atleast partially represent software and/or firmware for the processor.Non-limiting examples of the memory are a RAM or ROM that is accessibleby the processor.

The apparatuses according to the first aspect of the present inventionmay for instance be comprised in the first-type device, in one of theone or more second-type devices or in another device, such as forinstance a unit that implements a service (e.g. a server), such as forinstance a location service.

According to the first aspect of the present invention, a locationand/or an orientation of a first-type device is estimated at least basedon respective location estimates for one or more signal sources andrespective position relationships between the one or more signal sourcesand the first-type device. This estimated location and/or orientationmay for instance be of interest if the first-type device is connected inan ad-hoc network for multi-view audiovisual content capture, or may beused in other location/orientation-based services like navigation oraugmented reality, to name but a few non-limiting examples.

Therein, the location may for instance be expressed in terms of apre-defined coordinate system (e.g. in geodetic coordinates). Theorientation may for instance be understood as the arrangement of thefirst-type device (for instance the arrangement of one of its axes orplanes) with respect to a pre-defined coordinate system, which may forinstance be the same coordinate system in which the location isexpressed, or a different coordinate system.

The first-type device and the one or more second-type devices may forinstance be portable electronic devices, such as for instance mobilephones, personal digital assistants or media-rendering devices. Anon-limiting example of a first-type device is a “basic” device that isnot equipped with accurate positioning functionality (such as forinstance satellite-based positioning functionality). A non-limitingexample of a second-type device is a “sophisticated” device that isequipped with accurate positioning functionality (such as for instancesatellite-based positioning functionality).

Estimation of the location and/or orientation exploits one or moresignal sources that emit respective signals that can be received at thesecond-type devices (and, in some embodiments of the present invention,also at the first-type device). Non-limiting examples of such signalsare acoustic, optic, electric, magnetic or electromagnetic signals, toname but a few non-limiting examples, and the signal sources may then beembodied accordingly to be able to produce such signals.

The signal sources may be deployed in the environment in which thefirst-type and second-type devices are located on purpose (for instanceaccording to a specific plan), or may be present in the environmentanyway. In some embodiments of the present invention, one or more signalsources may also be comprised in the first-type device. However, some orall signal sources may also be not comprised in the first-type deviceand may also not be connected to or dependent on the first-type device.For instance, one signal source may be comprised in the first-typedevice, and the other signal source may not be comprised in thefirst-type device.

It should be noted that, even when stemming from the same signal source(and thus being identical when being transmitted by the signal source),signals from a signal source respectively received at different devices(e.g. at the first-type device and the second-type device, or atdifferent second-type devices may differ, for instance due to differentreception times and/or different signal propagation conditions towardsthe different devices (including different propagation delays).

Estimation of the location and/or orientation is based on respectivelocation estimates for the one or more signal sources (i.e. there is forinstance one location estimate per signal source) and respectiveposition relationships between the one or more signals sources and thefirst-type device (i.e. there is for instance one position relationshipbetween each signal source and the first-type device). Consequently,estimation may be based on a single signal source only, or on two ormore signal sources.

The location estimates for the one or more signal sources may forinstance at least partially be expressed as a region or area or also ina statistical sense, for instance by a probability density function. Alocation estimate for a signal source is understood throughout thisspecification as an estimate of the location of a signal source.

At least one location estimate for a signal source is determined atleast based on respective signals from a signal source received at oneor more second-type devices (i.e. for instance one signal transmittedfrom the signal source is received at each second-type device, thusyielding a respective received signal per second-type device, whereinthe received signals at different second-type devices can be the same ordifferent (for instance differently delayed)) and respective locationsof the one or more second-type devices (i.e. there is for instance onelocation per second-type device).

The location of the second-type device may for instance be determined byusing a locating technique, such as for instance a satellite-basedpositioning technique (e.g. according to a Global Navigation SatelliteSystem (GNSS) such as for instance the Global Positioning System (GPS),Galileo or the GLONASS system), a cell-ID-based positioning system or apositioning system that is based on reception of beacon (e.g. basestation) signals and according triangulation, to name but a fewnon-limiting examples. The location of the second-type device mayequally well be known a priori, for instance in case that thesecond-type device is fixed (non-moving). The location may then forinstance be considered to have been determined only once, for instanceduring the installation of the second-type device. The location of thesecond-type device may be determined by the second-type device itself,or by another device (such as for instance a server that is involved inthe estimation of the location and/or orientation of the first-typedevice).

A location estimate for a signal source may for instance be determinedby estimating a direction and/or a distance between a second-type deviceand the signal source and combining this estimated direction and/ordistance and the location of the second-type device. The direction mayfor instance be estimated based on the signal received at thesecond-type device. The distance may also be estimated based on thesignal received at the second-type device, but may equally well beestimated as a fixed (e.g. pre-defined) value, for instance as a maximumpossible distance with respect to a sensitivity of the one or moresensors used for receiving the signal. In any case, the locationestimate for the signal source may be considered to be determined atleast based on the signal of the signal source received at thesecond-type device because the signal source to which the locationestimate pertains (and that thus identifies the location estimate) mayonly be differentiated from other signal sources by its signal.

Therein, it should be noted that, in case of several signal sourcesemitting signals, the respective signals from these several signalsources are superposed at a receiving second-type device and may have tobe separated to allow estimation of the respective directions and/ordistances towards the signal sources. This separation may for instancebe jointly performed with the estimation, for instance by using anExpectation-Maximisation (EM) algorithm or any other type of suitedalgorithm.

If several second-type devices have received signals from the samesignal source, a location estimate for the signal source may be obtainedfor each second-type device, and these location estimates may then forinstance be combined (e.g. by forming cross-sections) into a singlelocation estimate for the signal source to be used for the estimating ofthe location and/or orientation of the first-type device.

The estimating of the location and/or orientation is also based onrespective position relationships between the one or more signal sourcesand the first-type device (i.e. there is for instance one positionrelationship with respect to the first-type device for each signalsource).

Therein, a position relationship may for instance be understood as arelationship that expresses a relative positioning/arrangement of twoobjects (in this case the signal source and the first-type device) withrespect to each other. Non-limiting examples of such a positionrelationship may for instance be a direction between two objects, or adistance between two objects, or a direction and a distance between twoobjects, wherein both the distance and the direction may representranges or may be expressed in a statistical sense, for instance asprobability density functions).

A position relationship may for instance be known a priori, for instancesince the signal source is comprised in the first-type device, so thatthe locations of both can for instance be assumed to be the same orsubstantially the same.

Alternatively, a position relationship may be estimated based on asignal from the signal source received at the first-type device, forinstance by estimating, based on the signal from the signal sourcereceived at the first-type device, a direction and/or a distance betweenthe first-type device and the signal source.

It is of course also possible that both a priori known positionrelationships and estimated position relationships are used, forinstance if only one signal source is comprised in the first-type device(and is thus associated with an a priori known position relationship)and one or more other signal sources are outside of the first-typedevice (and are thus associated with estimated position relationships).

At least based on the respective location estimates for the one or moresignal sources and the respective position relationships between the oneor more signal sources and the first-type device (which are either knowna priori or estimated), the position and/or orientation of thefirst-type device is estimated.

This estimation may for instance be performed by combining the locationestimates and the position relationships that pertain to the same signalsources to obtain signal-source-specific constraints (candidates) forthe location and/or orientation of the first-type device, and mergingthese signal-source-specific constraints to arrive at a final estimateof the location and/or orientation of the first-type device.

It should be noted that there may exist further signal sources than theone or more signal sources actually used for the estimation of thelocation and/or orientation of the first-type device.

The first aspect of the present invention thus may be considered toexploit that respective signals from one or more signal sources receivedby a second-type device (which is capable of accurately determining itslocation and/or orientation by itself), together with at least alocation of the second-type device determined by the second-type device,allow determining location estimates for the one or more signal sources(for instance by estimating directions and/or distances towards thesignal sources and combining these directions and/or distances withlocation of the second-type device). Based on these location estimates,and on position relationships (e.g. directions and/or distances) betweenthese signal sources and a first-type device (which is not capable ofaccurately determining its location and/or orientation by itself), thelocation and/or orientation of the first-type device can be estimated.The position relationships may be known a priori, for instance in caseswhere a signal source is part of the first-type device, and/or may beestimated from signals received by the first-type device from the one ormore signal sources. In any case, the first-type device usescollaboration with at least one second-type device so that an estimateof the location and/or orientation of the first-type device can beobtained. According to an embodiment of the first aspect of the presentinvention, the one or more signal sources are sound sources that emitrespective audio signals. The sound sources may for instance be humanbeings that produce sound by, for instance, talking. Other non-limitingexamples of sound sources are loudspeakers. The audio signals may forinstance be received at the second-type devices (and also at thefirst-type device, in case that the signal source is not comprised inthe first-type device) with one or more microphones. Using sound sourcesas signal sources may for instance be advantageous since capturingdirectional audio may be easier than capturing, for instance,directional radio signals. The frequencies of audio signals are lowerthan the frequencies of, for instance, radio signals, and as a result,the amount of data to analyse may be far less. Furthermore, the requiredantenna size might also be a problem for certain radio frequencies, incontrast to the comparably small microphones for audio capture.Furthermore, the audio environment and the sound sources are availablearound the first-type and second-type devices by nature, so that actualdeployment of sound sources for location and/or orientation estimationmay not be necessary. It may also not be necessary to trigger signalemission of such sound sources, since they may produce the sound signalson their own behalf (for instance if the sound sources are human beingsthat emit audio signals by talking).

According to an embodiment of the first aspect of the present invention,the at least one location estimate for the signal source is determinedat least based on the respective locations of the one or moresecond-type devices, respective orientations of the one or moresecond-type devices and respective estimates of respective directionsbetween the signal source and the one or more second-type devicesdetermined at least based on the respective signals received at the oneor more second-type devices. It may also be the case that more than oneor even all of the respective location estimates are determined in thesame way as the at least one location estimate.

An estimate of a direction between a signal source and a (single)second-type device may for instance be determined based on the signalfrom the signal source received at the second-type device if thesecond-type uses two or more sensors (e.g. microphones in case of anaudio signal) for receiving the signal. An estimate of the direction ofthe signal source relative to the array of two or more sensors then maybe calculated based on the delays of the reception outputs of the sensorarray caused by the received signal, the geometry (spacing) of thesensor array and the signal propagation speed. This (relative) directionmay then be transformed into an absolute direction by considering theorientation of the second-type device (which may for instance bedetermined from a magnetometer or any other type of compass). Thisabsolute direction may then be combined with the location of thesecond-type device as an anchor point to form a ray that then representsthe location estimate. Therein, the ray may for instance also berepresented by a coil or sector, for instance to account for estimationerrors, and then this coil or sector may represent the locationestimate.

If signals of the same signal source have been received at severalsecond-type devices, the respective location estimates of these severaldevices may for instance be combined to arrive at a single locationestimate for this signal source. An example for this combination oflocation estimates of several second-type devices may be averaging. Asanother example, basic triangulation could be used as a technique forcombining location estimates of several second-type devices.

If signals from more than one signal source are received (in superposedform) at the second-type device, it may be necessary to separate thesesignals before estimating the respective directions or to consider thepresence of multiple signals when estimating the respective directions.

The at least one location estimate for the signal source may forinstance further be based on respective estimates of respectivedistances between the signal source and the one or more second-typedevices.

A distance between a signal source and a second-type device may forinstance be estimated based on the power level of the signal from thesignal source received at the second-type device. Therein, it may beadvantageous that the transmission power of the signal source (or atleast a part thereof) is known or can be estimated. A transmission powerof a signal source may for instance be known if the signal source iscomprised in the first-type device. If signals from multiple signalsources are jointly received at the second-type device, accordingseparation of the signals may have to be performed before or duringestimation of the distance. A maximum distance of a signal source mayalso be derived from the sensitivity of the one or more sensors used toreceive the signal from the signal source. The estimated distance (orestimated maximum possible distance) between the signal source and thesecond-type device may then be used to reduce the direction-basedlocation estimate described in the previous embodiment. For instance, ifthe direction-based location estimate has the form of a ray, theestimated direction would allow reducing the location estimate to apoint on this ray. However, also the estimated distance may berepresented by a certain range to account for estimation errors.

According to an embodiment of the first aspect of the present invention,the at least one location estimate for the signal source is determinedfurther based on at least one previously determined location estimatefor the signal source. For instance, instead of using a currentlydetermined location estimate as a result, a cross-section of thepreviously determined location estimate and the currently determinedlocation estimate could be used as a result. This may for instance beadvantageous if the signal sources are known or can be assumed to benon-moving or if the frequency at which new location estimates for thesignal sources are determined is large enough so that a signal sourcecan be assumed to have only insignificantly moved between two locationestimates.

According to an embodiment of the first aspect of the present invention,the at least one location estimate for the signal source is determinedfurther based on level differences between respective signals from atleast two of the one or more signal sources received at a second-typedevice of the one or more second-type devices. The level differences mayfor instance be differences in the respective received power levels ofsignals received from the at least two signal sources. Large leveldifferences of signals received from two signal sources may for instanceindicate that one of these signal sources has to be farther apart fromthe location of the second-type device than the other signal source.

According to an embodiment of the first aspect of the present invention,the respective signals received at the second-type devices are analysedto decide if they stem from the same signal source and are thus jointlyuseable as a basis for determining the at least one location estimatefor the signal source. The analysis may for instance comprise comparingthe signals received at different second-type devices, for instance bycross-correlation. In this comparison, time shifts of the receivedsignals which may for instance be due to different signal propagationdelays may be taken into account. If signals from several signal sourcesare received at each second-type device, these signals may be separatedbefore conducting the analysis.

Combining (for instance by averaging or by forming cross-sections)location estimates for the same signal source from two or moresecond-type devices may significantly improve the quality of thislocation estimate and thus also the quality of the estimate of thelocation and/or orientation of the first-type device that depends onthis location estimate.

According to an embodiment of the first aspect of the present invention,at least one of the one or more signal sources is comprised in thefirst-type device, and a position relationship between the at least onesignal source comprised in the first-type device and the first-typedevice is known a priori (e.g. without a need for estimation), forinstance at least at the first-type device. The signal source comprisedin the first-type device may for instance be the only signal source usedfor estimating the location and/or orientation of the first-type device.Having a signal source in the first-type device may account for caseswhere no external signal sources, i.e. signal sources that are notcomprised in the first-type device, are present and thus ensures thatthe location and/or orientation of the first-type device can beestimated even if (at least temporarily) there are no external signalsources present. Equally well, this estimating may be further based onone or more further signal sources that are not comprised in thefirst-type device. The signal emitted by the signal source comprised inthe first-type device may then be received by the one or moresecond-type devices, and this one or more respective received signalsthen form an at least partial basis for the determining of the locationestimate for this signal source. Since this signal source is known to becomprised in the first-type device, the location estimate then may beconsidered as an estimate of the location of the first-type device.

The estimating of the at least one of the location and the orientationof the first-type device may then for instance be performed by thefirst-type device. The signal source in the first-type device may thenfor instance emit a signal that is received by the one or moresecond-type devices. The signal emitted by the signal source comprisedin the first-type device may for instance be optimized to allow properestimation of the direction and/or distance towards the first-typedevice by the second-type devices receiving this signal. The signalemitted by the signal source comprised in the first-type device may forinstance be emitted with a pre-defined transmission power level to allowestimation of the distance between the second-type device that receivesthis signal and the first-type device, for instance based on acomparison of the power level of the received signal and the pre-definedtransmission power level. Using such a pre-defined power level may forinstance be advantageous if the distance between the signal sourcecomprised in the first-type device and the second-type device isestimated by the second-type device. Alternatively, if the distance isestimated by the first-type device, for instance based on information onthe signal (of the signal source comprised in the first-type device)received at the second-type device, any transmission power level may beused; it may then be sufficient that the unit of the first-type devicethat performs the estimation of the distance is provided withinformation on the transmission power level that was used at the signalsource for emission of the signal.

At the first-type device, then either the respective location estimatesfor the one or more signal sources or information based on which therespective location estimates (e.g. directions and/or distances) for theone or more signal sources are derivable may be received from at leastone of the one or more second-type devices and a service to which theone or more second-type devices provided one of the respective locationestimates for the one or more signal sources and information based onwhich the respective location estimates for the one or more signalsources are derivable.

In the example case that the signal source comprised in the first-typedevice is the only signal source from which the one or more second-typedevices received a signal, the location estimates pertaining to thissignal source then represent a location estimate for the first-typedevice. If the one or more second-type device and the first-type devicereceived respective signals also from one or more further signal sources(not comprised in the first-type device), information on these receivedsignals may additionally be used by the first-type device to estimatethe location and/or orientation of the first-type device.

According to an embodiment of the present invention, at least oneposition relationship between a signal source of the one or more signalsources and the first-type device is estimated at least based on asignal from the signal source received at the first-type device.

The position relationship may for instance be estimated based on thesignal from the signal source received at the first-type device byestimating a direction and/or a distance between the first-type deviceand the signal source. The direction may for instance be estimated basedon the signal received at the first-type device. The distance may alsobe estimated based on the signal received at the first-type device, butmay equally well be estimated as a fixed (e.g. pre-defined) value, suchas for instance as the maximum possible distance, for instance withrespect to the sensitivity of the one or more sensors used for receivingthe signal. The position relationship may then be considered to beestimated at least based on the signal of the signal source received atthe first-type device, because the signal source to which the estimateof the position relationship pertains (and that thus identifies theestimate of the position relationship) may only be differentiated fromother signal sources by its signal. Therein, it should again be notedthat in case of several signal sources emitting signals, the respectivesignals from these several signal sources are superposed at thereceiving first-type device and may have to be separated to allowestimation of the respective directions and/or distances of the signalsources. This separation may for instance be jointly performed with theestimation, for instance by using an Expectation-Maximisation (EM)algorithm or any other type of suited algorithm.

The at least one position relationship between the signal source and thefirst-type device may for instance be estimated at least based on anestimate of a direction between the signal source and the first-typedevice determined at least based on the signal received at thefirst-type device. It may also be the case that more than one or evenall of the respective estimates of the position relationships aredetermined in the same way as the at least one estimate of the positionrelationship.

An estimate of a direction between a signal source and a first-typedevice may for instance be determined based on the signal from thesignal source received at the first-type device if the first-type deviceuses two or more sensors (e.g. microphones in case of an audio signal)for receiving the signal. An estimate of the direction of the signalsource relative to the array of two or more sensors then may becalculated based on the delays of the outputs of the sensor array causedby the received signal, the geometry (spacing) of the sensor array andthe signal propagation speed. This relative direction may be transformedinto an absolute direction by considering the orientation of thefirst-type device, if this orientation is available (for instance if itcan be determined from a magnetometer or any other type of compass).This absolute direction then constitutes the estimated positionrelationship. Equally well, the relative direction may constitute theestimated position relationship. Relative directions may for instancealready be valuable for the estimation of the location and/ororientation of the first-type device if respective relative directionsbetween the first-type device and at least two signal sources areavailable, because both relative directions refer to the same sensorarray and are thus fixed with respect to each other.

Once again, if signals from more than one signal source are received (insuperposed form) at the first-type device, it may be necessary toseparate these signals before estimating the respective directions or toconsider the presence of multiple signals when estimating the respectivedirections.

The at least one position relationship between the signal source and thefirst-type device may for instance be estimated further based on anorientation of the first-type device. Knowledge on the orientation ofthe first-type device (which is for instance obtainable from a compass)allows transforming relative directions that can be estimated based on asignal from a signal source received at a sensor array of the first-typedevice relative to the sensor array into an absolute direction, whichthen may constitute the estimated position relationship. Even if only alocation estimate for a single signal source is available, such anabsolute direction (as estimate of the position relationship) betweenthis signal source and the first-type device may already yield anaccurate estimate of the location of the first-type device.

The at least one position relationship between the signal source and thefirst-type device may for instance be estimated further based on anestimate of a distance between the signal source and the first-typedevice. Consideration of the distance may even further improve thequality of the estimate of the location and/or orientation of thefirst-type device. The distance may for instance be estimated from thesignal received at the first-type device (for instance based on thereceived power level thereof). In case of multiple signal sources,according separation of their signals received at the first-type devicemay be necessary before the estimation process or consideration of thefact that there are several signal sources in the estimation process.Equally well, the estimate of the distance may be estimated as a fixed(e.g. pre-defined) value, such as for instance as the maximum possibledistance, for instance with respect to the sensitivity of the one ormore sensors used for receiving the signal.

The estimating of the at least one of the location and the orientationof the first-type device may further be based on level differencesbetween respective signals from at least two of the one or more signalsources received at the first-type device. The level differences may forinstance be differences in the respective received power levels ofsignals received from the at least two signal sources. Such informationmay be used as a further constraint for the estimating of the at leastone of the location and the orientation of the first-type device.Exploiting knowledge on level differences is for instance advantageousin situations where only relative directions from the first-type deviceto two or more signal sources (with known locations) are known, sincethe ambiguity in the location and/or orientation of the first-typedevice may then be resolvable.

Signal reception at the first-type device and the one or moresecond-type devices may for instance be one of continued and repeateduntil a pre-defined number of signal sources is considered to be presentas a basis for the estimating of the at least one of the location andthe orientation of the first-type device or until a pre-defined accuracyof the estimation of the at least one of the location and theorientation of the first-type device has been achieved. Reception ofsignals from the one or more signal sources is thus either continued orrepeated until either a pre-defined number of signal sources isconsidered to be present or until a pre-defined estimation accuracy isreached. Further continuation or repetition of the signal reception mayfor instance be useful if the signal sources were not transmittingbefore or if not enough signal sources were transmitting before.

The estimating of the at least one of the location and the orientationof the first-type device may for instance be performed by the first-typedevice.

This may further comprise receiving, at the first-type device,respective signals from the one or more signal sources, determining, atthe first-type device, the respective estimates of the respectiveposition relationships between the one or more signal sources and thefirst-type device based on the received signals, and receiving, at thefirst-type device, one of the respective location estimates for the oneor more signal sources and information based on which the respectivelocation estimates for the one or more signal sources are derivable fromat least one of the one or more second-type devices and a service towhich the one or more second-type devices provided one of the respectivelocation estimates for the one or more signal sources and informationbased on which the respective location estimates for the one or moresignal sources are derivable. The service may for instance be executedby an apparatus, such as for instance a server. The information based onwhich the respective location estimates for the one or more signalsources are derivable may for instance be the respective signalsreceived by the second-type devices (wherein in case of signals frommultiple signal sources being received at a second-type device, thesuperposition of these signals may be sent to the first-type device) andthe respective locations of the second-type devices. Instead of therespective signals received by the second-type devices, also aparameterization of these signals may be sent to the first-type device.

Some or all of the information received by the first-type device may forinstance have been scrambled for security and/or privacy reasons (whichmay be particularly important if audio signals are received).

According to an embodiment of the first aspect of the present invention,the estimating of the at least one of the location and the orientationof the first-type device is performed by the second-type device.

This may comprise receiving, at the second-type device, one of therespective position relationships between the one or more signal sourcesand the first-type device and information based on which the respectiveposition relationships between the one or more signal sources and thefirst-type device are derivable, obtaining, at the second-type device, alocation of the second-type device, receiving, at the second-typedevice, respective signals from one or more signal sources; anddetermining, at the second-type device, respective location estimatesfor the one or more signal sources at least based on the receivedsignals from the one or more signal sources and the obtained location ofthe second-type device.

The information based on which the respective position relationshipsbetween the one or more signal sources and the first-type device arederivable may for instance be the respective signals received by thefirst-type device from the one or more signal sources (wherein in caseof signals from multiple signal sources being received at the first-typedevice, the superposition of these signals may be sent to the service).Instead of the respective signals received by the first-type device,also a parameterization of these signals may be sent to the service.

The estimating of the at least one of the location and the orientationof the first-type device at the second-type device may further comprisereceiving, at the second-type device, one of respective locationestimates for the one or more signal sources and information based onwhich the respective location estimates for the one or more signalsources are derivable from at least one of at least one othersecond-type device of the one or more second-type devices and a serviceto which the at least one other second-type device provided one of therespective location estimates for the one or more signal sources and theinformation based on which the respective location estimates for the oneor more signal sources are derivable, and the respective locationestimates are then determined at the second-type device further based onthe received one of the respective location estimates for the one ormore signal sources and the information based on which the respectivelocation estimates for the one or more signal sources are derivable.

Some or all of the information received by the second-type device mayfor instance have been scrambled for security and/or privacy reasons(which may be particularly important if audio signals are received).

According to an embodiment of the first aspect of the present invention,the estimating of the at least one of the location and the orientationof the first-type device is performed by a service, for instance alocation service. The service may for instance be executed by anapparatus, such as for instance a server.

This may further comprise receiving, by the service, one of therespective location estimates for the one or more signal sources andinformation based on which the respective location estimates for the oneor more signal sources are derivable from the one or more second-typedevices, and receiving, by the service, one of the respective positionrelationships between the one or more signal sources and the first-typedevice and information based on which the respective positionrelationships between the one or more signal sources and the first-typedevice are derivable.

The information based on which the respective location estimates for theone or more signal sources are derivable may for instance be therespective signals received by the second-type devices (wherein in caseof signals from multiple signal sources being received at a second-typedevice, the superposition of these signals may be sent to the service)and the respective locations of the second-type devices. Instead of therespective signals received by the second-type devices, also aparameterization of these signals may be sent to the service.

The information based on which the respective position relationshipsbetween the one or more signal sources and the first-type device arederivable may for instance be the respective signals received by thefirst-type device from the one or more signal sources (wherein in caseof signals from multiple signal sources being received at the first-typedevice, the superposition of these signals may be sent to the service).Instead of the respective signals received by the first-type device,also a parameterization of these signals may be sent to the service.

Some or all of the information received by the service may for instancehave been scrambled for security and/or privacy reasons (which may beparticularly important if audio signals are received).

According to an embodiment of the first aspect of the present invention,the estimating of the at least one of the location and the orientationof the first-type device comprises considering at least those of therespective location estimates and the respective position relationshipsthat pertain to the same signal sources of the one or more signalsources for setting up signal-source-specific constraints for the atleast one of the location and the orientation of the first-type device,and merging at least the signal-source-specific constraints to obtain anestimate of the at least one of the location and the orientation of thefirst-type device.

The respective location estimates and the respective positionrelationships that pertain to the same signal sources may for instancebe identified based on an analysis (e.g. a comparison, for instance byforming cross-correlations) of the respective signals emitted by the oneor more signal sources (and received at the first-type device and theone or more second-type devices).

According to an embodiment of the first aspect of the present invention,the estimating of the at least one of the location and the orientationof the first-type device is further based on an estimated location ofthe first-type device. The estimated location may for instance have beenestimated by the first-type device. The estimated location may forinstance be estimated by using a cell-ID-based location technique in acellular communication system (e.g. a technique in which the first-typedevice knows the ID of the cell it is currently associated with an alsoknows the position and extension of the cell, without however knowingwhere exactly in this cell the first-type device is currently located).The estimated location may for instance have been estimated by using alocation technique that achieves coarser estimation accuracy than asatellite-based location technique. The estimated location may forinstance be determined based on a non-satellite-based locationtechnique. The estimated location may for instance be used as a furtherconstraint for the estimating of the at least one of the location andthe orientation of the first-type device.

According to an embodiment of the first aspect of the present invention,the estimating of the at least one of the location and the orientationof the first-type device is further based on at least one of apreviously estimated location and a previously estimated orientation.These previous estimates may for instance stem from a previous step ofestimating the at least one of the location and the orientation of thefirst-type device. For instance, instead of using a currently determinedlocation estimate as a result, a cross-section of the previouslydetermined location estimate and the currently determined locationestimate could be used as a result. This may for instance beadvantageous if the first-type device is known or can be assumed to benon-moving or if the frequency at which new location estimates for thefirst-type device are determined is large enough so that the first-typedevice can be assumed to have only insignificantly moved between twolocation estimates.

According to an embodiment of the first aspect of the present invention,wherein the first-type device triggers signal reception at the one ormore second-type devices in one of a direct manner and an indirectmanner. The signal reception (of the signals from the signal sources)may for instance be triggered indirectly by sending a request to aservice that then triggers the one or more second-type devices, ordirectly by sending (e.g. broadcasting) a request to the one or moresecond-type devices (for instance to those with the same cell-ID).

According to an embodiment of the first aspect of the present invention,in case that it is determined that, among one or more available locationestimates for one or more signal sources and among available positionrelationships between one or more signal sources and the first-typedevice, not even one of the available location estimates and one of theavailable position relationship pertain to the same signal source,action is taken to trigger signal reception at at least one second-typedevice that has not yet contributed to the available location estimates.Involving a further second-type device into the estimation process mayfor instance be useful if—due to lack of match between the availablelocation estimates and the available position relationship estimates—itis considered that the first-type device and the second-type devicesalready involved are not in the same environment. The furthersecond-type device then may for instance be selected to be in aneighborhood (for instance in a neighboring cell in case of a cellularsystem).

In a second aspect of the present invention, a method is disclosed, themethod comprising receiving, at a first-type device, a signal from asignal source of one or more signal sources to serve as an at leastpartial basis for estimating a position relationship between the signalsource and the first-type device, wherein an estimate of at least one ofa location and an orientation of the first-type device is derivable atleast based on respective position relationships between the one or moresignal sources and the first-type device and respective locationestimates for the one or more signal sources, and wherein at least onelocation estimate for a signal source of the one or more signal sourcesis determinable at least based on respective signals from the signalsource received at one or more second-type devices and respectivelocations of the one or more second-type devices.

In this second aspect of the present invention, furthermore a computerprogram is disclosed, comprising program code for performing the methodaccording to the second aspect of the present invention when thecomputer program is executed on a processor. The computer program mayfor instance be distributable via a network, such as for instance theInternet. The computer program may for instance be storable or encodablein a computer-readable medium. The computer program may for instance atleast partially represent software and/or firmware of the processor.

In this second aspect of the present invention, furthermore acomputer-readable medium is disclosed, having a computer programaccording to the second aspect of the present invention stored thereon.The computer-readable medium may for instance be embodied as anelectric, magnetic, electro-magnetic, optic or other storage medium, andmay either be a removable medium or a medium that is fixedly installedin an apparatus or device. Non-limiting examples of such acomputer-readable medium are a Random-Access Memory (RAM) or a Read-OnlyMemory (ROM). The computer-readable medium may for instance be atangible medium, for instance a tangible storage medium. Acomputer-readable medium is understood to be readable by a computer,such as for instance a processor.

In this second aspect of the present invention, furthermore an apparatusis disclosed, configured to perform the method according to the secondaspect of the present invention.

In this second aspect of the present invention, furthermore an apparatusis disclosed, comprising means for receiving, at a first-type device, asignal from a signal source of one or more signal sources to serve as anat least partial basis for estimating a position relationship betweenthe signal source and the first-type device, wherein an estimate of atleast one of a location and an orientation of the first-type device isderivable at least based on respective position relationships betweenthe one or more signal sources and the first-type device and respectivelocation estimates for the one or more signal sources, and wherein atleast one location estimate for a signal source of the one or moresignal sources is determinable at least based on respective signals fromthe signal source received at one or more second-type devices andrespective locations of the one or more second-type devices.

In this second aspect of the present invention, furthermore an apparatusis disclosed, comprising at least one processor; and at least one memoryincluding computer program code, the at least one memory and thecomputer program code configured to, with the at least one processor,cause the apparatus at least to receive, at a first-type device, asignal from a signal source of one or more signal sources to serve as anat least partial basis for estimating a position relationship betweenthe signal source and the first-type device, wherein an estimate of atleast one of a location and an orientation of the first-type device isderivable at least based on respective position relationships betweenthe one or more signal sources and the first-type device and respectivelocation estimates for the one or more signal sources, and wherein atleast one location estimate for a signal source of the one or moresignal sources is determinable at least based on respective signals fromthe signal source received at one or more second-type devices andrespective locations of the one or more second-type devices.

For this second aspect of the present invention, the above descriptionof the first aspect of the present invention and of its embodimentsequally applies. In particular, all features and advantages of the firstaspect of the present invention (including its embodiments) shall beunderstood to be disclosed in connection with the second aspect of thepresent invention as well. Therein, the estimated position relationshipis one of the respective position relationships between the one or moresignal sources and the first-type device based on which the estimate ofthe location and/or orientation of the first-type device is derivable.

According to an embodiment of the second aspect of the presentinvention, information related to the signal received at the first-typedevice is communicated from the first-type device to one of a serviceand a second-type device of the second-type devices, wherein the one ofthe service and the second-type device is configured to estimate the atleast one of the location and the orientation of the first-type devicebased on the information related to the signal received at thefirst-type device and the respective location estimates for the one ormore signal sources. The information related to the signal received atthe first-type device may for instance be the received signal itself, ora sum signal that contains this received signal and furthermorerespective signals from further signal sources received at thefirst-type device, or a parameterization of the received signal, or aparameterization of some or all signals contained in the sum signal, toname but a few non-limiting examples. The service may for instance beexecuted by an apparatus, such as for instance a server.

In a third aspect of the present invention, a method is disclosed,comprising receiving, at a second-type device, a signal from a signalsource of one or more signal sources to serve, together with at least alocation of the second-type device, as an at least partial basis fordetermining a location estimate for the signal source, wherein anestimate of at least one of a location and an orientation of afirst-type device is derivable at least based on respective locationestimates for the one or more signal sources, and respective positionrelationships between the one or more signal sources and the first-typedevice.

In this third aspect of the present invention, furthermore a computerprogram is disclosed, comprising program code for performing the methodaccording to the third aspect of the present invention when the computerprogram is executed on a processor. The computer program may forinstance be distributable via a network, such as for instance theInternet. The computer program may for instance be storable or encodablein a computer-readable medium. The computer program may for instance atleast partially represent software and/or firmware of the processor.

In this third aspect of the present invention, furthermore acomputer-readable medium is disclosed, having a computer programaccording to the third aspect of the present invention stored thereon.The computer-readable medium may for instance be embodied as anelectric, magnetic, electro-magnetic, optic or other storage medium, andmay either be a removable medium or a medium that is fixedly installedin an apparatus or device. Non-limiting examples of such acomputer-readable medium are a Random-Access Memory (RAM) or a Read-OnlyMemory (ROM). The computer-readable medium may for instance be atangible medium, for instance a tangible storage medium. Acomputer-readable medium is understood to be readable by a computer,such as for instance a processor.

In this third aspect of the present invention, furthermore an apparatusis disclosed, configured to perform the method according to the thirdaspect of the present invention.

In this third aspect of the present invention, furthermore an apparatusis disclosed, comprising means for receiving, at a second-type device, asignal from a signal source of one or more signal sources to serve,together with at least a location of the second-type device, as an atleast partial basis for determining a location estimate for the signalsource, wherein an estimate of at least one of a location and anorientation of a first-type device is derivable at least based onrespective location estimates for the one or more signal sources, andrespective position relationships between the one or more signal sourcesand the first-type device.

In this third aspect of the present invention, furthermore an apparatusis disclosed, comprising at least one processor; and at least one memoryincluding computer program code, the at least one memory and thecomputer program code configured to, with the at least one processor,cause the apparatus at least to receive, at a second-type device, asignal from a signal source of one or more signal sources to serve,together with at least a location of the second-type device, as an atleast partial basis for determining a location estimate for the signalsource, wherein an estimate of at least one of a location and anorientation of a first-type device is derivable at least based onrespective location estimates for the one or more signal sources, andrespective position relationships between the one or more signal sourcesand the first-type device.

For this third aspect of the present invention, the above description ofthe first and second aspects of the present invention and of theirembodiments equally applies. In particular, all features and advantagesof the first and second aspect of the present invention (including theirembodiments) shall be understood to be disclosed in connection with thethird aspect of the present invention as well. In particular, at leastone of the position relationships may for instance be known a priori(for instance if the related signal source is comprised in thefirst-type device) or may be estimated at least based on a signal fromthe related signal source received at the first-type device.

According to an embodiment of the third aspect of the present invention,information related to the signal received at the second-type device iscommunicated from the second-type device to one of a service, thefirst-type device and another second-type device, wherein the one of theservice, the first-type device and the other second-type device isconfigured to estimate the at least one of the location and theorientation of the first-type device based on the information related tothe signal received at the second-type device and the respectiveposition relationships between the one or more signal sources and thefirst-type device. The information related to the signal received at thesecond-type device may for instance be the received signal itself, or asum signal that contains this received signal and furthermore respectivesignals from further signal sources received at the second-type device,or a parameterization of the received signal, or a parameterization ofsome or all signals contained in the sum signal, to name but a fewnon-limiting examples. The service may for instance be executed by anapparatus, such as for instance a server.

Furthermore, by the second-type device, the location (and optionally theorientation) of the second-type device may be obtained, and location(and optionally the orientation) may be communicated to the one of theservice and the first-type device. Alternatively, the location of thesecond-type device may for instance be determined by the service.

In a fourth aspect of the present invention, a system is disclosed, thesystem comprising a first-type device, one or more second-type devices,and a processor comprised in one of the first-type device, one of theone or more second-type devices and a third-type device, the processorconfigured at least to estimate at least one of a location and anorientation of the first-type device at least based on

-   -   respective location estimates for one or more signal sources,        wherein at least one location estimate for a signal source of        the one or more signal sources is determined at least based on        respective signals from the signal source received at the one or        more second-type devices and respective locations of the one or        more second-type devices, and    -   respective position relationships between the one or more signal        sources and the first-type device.

For this fourth aspect of the present invention, the above descriptionof the first, second and third aspect of the present invention and oftheir embodiments equally applies. In particular, all features andadvantages of the first, second and third aspect of the presentinvention (including their embodiments) shall be understood to bedisclosed in connection with the fourth aspect of the present inventionas well. In particular, at least one of the position relationships mayfor instance be known a priori (for instance if the related signalsource is comprised in the first-type device) or may be estimated atleast based on a signal from the related signal source received at thefirst-type device.

The third-type device may for instance be a device that executes aservice, such as for instance a server.

It is to be noted that the above-described embodiments of the presentinvention are to be understood as non-limiting examples only.

Furthermore, the embodiments described above and in particular theirsingle features shall be understood to be disclosed in all possiblecombinations with each other.

These and further concepts of the invention will be apparent from andelucidated with reference to the detailed description presentedhereinafter.

BRIEF DESCRIPTION OF THE FIGURES

In the figures show:

FIG. 1 a: A schematic illustration of a system for location and/ororientation estimation according to an embodiment of the presentinvention;

FIG. 1 b: a schematic illustration a system for location and/ororientation estimation according to a further embodiment of the presentinvention;

FIG. 1 c: a schematic illustration a system for location and/ororientation estimation according to a further embodiment of the presentinvention;

FIG. 2: a schematic illustration of an apparatus comprised in afirst-type device of a system for location and/or orientation estimationaccording to an embodiment of the present invention;

FIG. 3: a schematic illustration of an apparatus comprised in asecond-type device of a system for location and/or orientationestimation according to an embodiment of the present invention;

FIG. 4: a schematic illustration of an apparatus comprised in a serverof a system for location and/or orientation estimation according to anembodiment of the present invention;

FIG. 5: a schematic illustration of a tangible storage medium accordingto an embodiment of the present invention;

FIG. 6: a flowchart of a method performed by a first-type device of asystem for location and/or orientation estimation according to anembodiment of the present invention;

FIG. 7: a flowchart of a method performed by a second-type device of asystem for location and/or orientation estimation according to anembodiment of the present invention;

FIG. 8 a: a flowchart of a method performed by a server of a system forlocation and/or orientation estimation according to an embodiment of thepresent invention;

FIG. 8 b: a flowchart of a method performed by a first-type device of asystem for location and/or orientation estimation according to anembodiment of the present invention;

FIG. 8 c: a flowchart of a method performed by a second-type device of asystem for location and/or orientation estimation according to anembodiment of the present invention;

FIG. 8 d: a flowchart of a method performed by a first-type device of asystem for location and/or orientation estimation according to anembodiment of the present invention;

FIG. 9: a schematic illustration of a delay difference cause by signalreception with a microphone array;

FIG. 10: a schematic illustration of the Inter channel Level Difference(ILD) and Inter channel Time Difference (ITD) parameters in binaural cuecoding;

FIG. 11: a schematic illustration of a reception of signals from asingle sound source by a first-type device and a second-type deviceaccording to an embodiment of the present invention;

FIG. 12: a schematic illustration of an example for estimating thelocation of the first-type device of FIG. 11 according to an embodimentof the present invention in case that the orientation of the first-typedevice is known;

FIG. 13: a schematic illustration of a reception of signals from twosound sources by a first-type device and a second-type device accordingto an embodiment of the present invention;

FIG. 14: a schematic illustration of an example for estimating thelocation of the first-type device of FIG. 13 according to an embodimentof the present invention in case that the orientation of the first-typedevice is known;

FIG. 15: a schematic illustration of an example for estimating thelocation of the first-type device of FIG. 13 according to an embodimentof the present invention in case that the orientation of the first-typedevice is not known;

FIG. 16: a schematic illustration of an example for estimating thelocation of the first-type device of FIG. 13 according to an embodimentof the present invention in case that the orientation of the first-typedevice is known and relative loudness of the sound sources isconsidered;

FIG. 17: a flowchart for estimating a location and/or an orientation ofa first-type device according to an embodiment of the present invention;

FIG. 18: a flowchart for estimating the location of sound sources (seestep 501 in FIG. 17) according to an embodiment of the presentinvention;

FIG. 19: a flowchart for estimating the location and/or orientation of afirst-type device based on the location of sound sources (see step 502in FIG. 17) according to an embodiment of the present invention; and

FIG. 20: a flowchart of a protocol for location and/or orientationestimation according to an embodiment of the present invention.

DETAILED DESCRIPTION OF EMBODIMENTS OF THE INVENTION

The present invention pertains to collaborative estimation of a locationand/or orientation of a first-type device (which is not capable ofaccurately determining its position by itself) using support from atleast a second-type device (which is capable of accurately determiningits position by itself).

In embodiments of the present invention, both the first-type device andthe at least one second-type device receive signals from one or moresignal sources. These received signals are analyzed to determineposition relationships between the signal sources and the devices.Additional knowledge on the location of the second-type device thenallows estimating the location and/or orientation of the first-typedevice.

In other embodiments of the present invention, the first-type devicecomprises at least one signal source, so that the position relationshipbetween this signal source and the first-type device is known a priori.A signal from this signal source is then received by the at least onesecond-type device and used to determine a location estimate for thissignal source, which, with the a priori known position relationshipbetween the first-type device and this signal source, yields a locationestimate for the first-type device. If further signal sources arepresent, which are not comprised in the first-type device (e.g. humanbeings acting as natural audio signal sources), signals from thesefurther signal sources may be received and analysed by both thefirst-type device (to estimate position relationships between thefirst-type device and these signal sources) and the second-type device(to determine location estimates for these signal sources) as a furtherbasis for the estimation of the location and/or orientation of thefirst-type device.

This concept will now be further elucidated with reference to FIGS. 1a-8 d. FIGS. 9-20 are dedicated to a specific application field of thepresent invention, where sound sources that are received by microphonesserve as signal sources. The description of all FIGS. 1-19 is explicitlyunderstood to be supported and supplemented by the above description inthe SUMMARY section of this specification.

FIG. 1 a is a schematic illustration of a system 1 for location and/ororientation estimation according to an embodiment of the presentinvention. System 1 comprises a signal source 10, a first-type device11, a second-type device 12 and a server (or in more general terms, athird-type device) 13.

In the system 1 of FIG. 1 a, it is assumed that the location and/ororientation of first-type device 11 is estimated by server 13 based oninformation received from first-type device 11 and second-type device12. This information pertains to signals from signal source 10 that arereceived at first-type device 11 and second-type device 12, andinformation on the location of second-type device 12. The server may forinstance execute a dedicated service linked to a multi-view audiovisualmedia capture application or service provision or linked to multi-userevent casting.

FIG. 1 b schematically illustrates an alternative system 1′ with asignal source 10 and a modified first-type device 11′ and a modifiedsecond-type device 12′. In the system 1′, the location and/ororientation of first-type device 11′ may be estimated by first-typedevice 11′, or by second-type device 12′, there is no server for thistask any more. If first-type device 11′ performs the estimation, it mayfor instance receive information on a signal received from signal source10 at second-type device 12′, and on a location of second-type device12′ determined by second-type device 12′, and process this informationtogether with the information on a signal from signal source 10 receivedat first-type device 11′ itself. If second-type device 12′ performs theestimation, it may for instance receive information on a signal receivedfrom signal source 10 at first-type device 11′, and process thisinformation together with the information on a signal from signal source10 received at second-type device 12′ and on a location of second-typedevice 12′ determined by second-type device 12′ itself.

In FIGS. 1 a and 1 b, for the sake of simplicity of presentation, only asingle signal source and a single second-type device are depicted. It ishowever to be understood that the present invention is equally wellintended for use of several signal sources and/or several second-typedevices. The case of several signal sources is for instance discussedwith reference to FIGS. 13-16 below.

FIG. 1 c schematically illustrates a further alternative system 1″ wherea signal source 10″ is comprised in modified first-type device 11″. Inthe system 1″, the location and/or orientation of first-type device 11″may be estimated by first-type device 11″, or by second-type device 12″.Signal source 10″ may for instance emit a signal that is received bysecond-type device 12″. This received signal or information thereon maythen be used to determine, together with a determined location and/ororientation of the second-type device 12″ or on information thereon, alocation estimate for signal source 10″. Since signal source 10″ iscomprised in first-type device 11″, the position relationship betweenthe first-type device 11″ and the signal source 10″ is known a priori(their locations may then for instance be known to be equal), and thelocation estimate for the signal source 10″ thus may be considered as alocation estimate for the first-type device 11″.

If the location and/or orientation of the first-type device 11″ isestimated by the second-type device 12″, it may be necessary to providethe second-type device 12″ with information on the a priori knownposition relationship between the first-type device 11″ and the signalsource 10″, since this knowledge may not be present at the second-typedevice 12″. This may for instance be achieved by including specificinformation into the signal emitted by signal source 10″, for instancean information sequence that is known to the second-type device 12″. Thesecond-type device 12″, when identifying this information sequence, maythen be aware that the signal source from which this informationsequence stems is comprised in the first-type device 11″ (while othersignal sources form which signals without such an information sequenceare received may then be considered to be not comprised in first-typedevice 11″). However, for instance in scenarios where the signal source10″ comprised in the first-type device 11″ is the only signal source, itmay implicitly be known or assumed by the second-type device 12″ thatsignal source 10″ is comprised in first-type device 11′.

If the location and/or orientation of the first-type device 11″ isdetermined by the first-type device 11″, it may be necessary that thefirst-type device 11″ receives information on the location estimate forsignal source 10″ determined by the second-type device 12″ orinformation based on which first-type device 11″ can determine such alocation estimate for signal source 10″. For instance, the signalreceived from signal source 10″ at the second-type device 12″ may beprovided at least in parts to the first-type device 11″. Since thesignal emitted by the signal source 10″ is known to the first-typedevice 11″, the first-type device 11″ may associate the signal receivedat the second-type device 12″ and provided to the first-type device 11″with signal source 10″, for instance by correlation. The first-typedevice 11″ is then aware that the location estimate for signal source10″ also represents a location estimate for the first-type device 11″.

In FIG. 1 c, it was assumed that only one signal source 10″ is present,and that this signal source is comprised in first-type device 11″. Itmay of course also be the case that signals from further signal sources,which may be internal and/or external to first-type device 11″, arereceived by the second-type device 12″ (and also, in case of externalsignal sources, by the first-type device 11″) and used as a furtherbasis for estimating the location and/or orientation of the first-typedevice 11″ as for instance explained with reference to FIG. 1 b above.

FIG. 2 is a schematic illustration of an apparatus 110 comprised in afirst-type device of a system for location and/or orientation estimationaccording to an embodiment of the present invention. This apparatus 110may be comprised in first-type device 11 in system 1 of FIG. 1 a, or maybe comprised in the modified first-type device 11′ of system 1′ of FIG.1 b, as will be described in further detail below.

Apparatus 110 comprises a processor 118 that executes program codestored in program memory 111. Main memory 112 is used by processor 118for instance as working memory. Processor 118 further interacts with oneor more optional sensors 113 (for instance microphones or antennas) thatare configured to receive signals from one or more signal sources. Suchsensors 113 may for instance not be required in embodiments of thepresent invention where estimation of the location and/or orientation ofthe first-type device is based on a single signal source that iscomprised in the first-type device (see FIG. 1 c). Processor 118 furtherinteracts with an optional user interface 115 that allows interactionbetween a user and first-type device 11/11′/11″. Further optionalcomponents processor 118 interacts with are a coarse location unit 116that allows coarsely locating first-type device 11/11′/11″ and anorientation unit 117, that allows determining an orientation offirst-type device 11/11′/11″. This orientation unit 117 may for instancebe embodied as a digital compass. Finally, processor 118 interacts witha communication interface 114, that allows communicating with otherdevices, for instance with server 13 of system 1 of FIG. 1 a, or withmodified second-type device 12′ of system 1′ of FIG. 1 b, or withmodified second-type device 12″ of system 2″ of FIG. 1 c.

The coarse location unit 116 of apparatus 110 is understood to allowonly a comparably rough positioning of the first-type device 11/11′/11″as compared to the positioning that is used by the second-type device12/12′/12″ (see the according description below). An exampleconstellation could for instance be no positioning or only a cell-IDbased positioning (without triangulation) being implemented by thefirst-type device 11/11′/11″ and a GNSS-based positioning beingimplemented by the second-type device 12/12′/12″.

Processor 118 further interacts with an optional signalling unit 119,which is configured to act as a signal source (e.g. as signal source 10″of System 1″ FIG. 1 c), i.e. to emit a signal. If the signal source is asound source, the signalling unit 119 may for instance be embodied as aloudspeaker that emits an audio signal, for instance when triggered byprocessor 118. The presence of a signalling unit 119 in apparatus 110may allow dispensing with external signal sources (i.e. signal sourcesthat are not comprised in a first-type device such as first-type device11″ of system 1″ of FIG. 1 c).

FIG. 3 is a schematic illustration of an apparatus 120 comprised in asecond-type device of a system for location and/or orientationestimation according to an embodiment of the present invention. Thisapparatus 120 may be comprised in second-type device 12 in system 1 ofFIG. 1 a, or may be comprised in the modified second-type device 12′ ofsystem 1′ of FIG. 1 b, or may be comprised in the modified second-typedevice 12″ of system 1″ of FIG. 1 c, as will be described in furtherdetail below.

Apparatus 120 comprises a processor 128 that executes program codestored in program memory 121. Main memory 122 is used by processor 128for instance as working memory. Processor 128 further interacts with oneor more sensors 123 (for instance microphones or antennas) that areconfigured to receive signals from one or more signal sources. Processor128 further interacts with an optional user interface 125 that allowsinteraction between a user and second-type device 12/12′/12″. Furthercomponents processor 128 interacts with are location unit 116, whichallows locating second-type device 12/12′/12″, and an optionalorientation unit 127, that allows determining an orientation ofsecond-type device 12/12′/12″. This orientation unit 127 may forinstance be embodied as a digital compass or use tracking methods (suchas for instance head tracking) to determine an orientation of thesecond-type device 12/12′/12″. Finally, processor 128 interacts with acommunication interface 124, that allows communicating with otherdevices, for instance with server 13 of system 1 of FIG. 1 a, or withmodified first-type device 11′ of system 1′ of FIG. 1 b or with modifiedfirst-type device 11″ of system 1″ of FIG. 1 c.

Location unit 126 may implement one or more positioning techniques forpositioning second-type device 12/12′/12″. A location of a device canfor instance be estimated using dedicated sensory information bymonitoring, e.g. by using a GNSS-based positioning system like GPS. Whena device is connected to a cellular network, the cell ID may provide abasic location information (wherein the granularity of positioning isdetermined by the cell size) that may be enhanced by further techniques.Knowledge of the cell ID in a cellular network corresponds to knowledgeof an ID of the present Wireless Local Area Network (WLAN) access point,wherein the coverage area of a WLAN access point may generally besmaller than a cell of a cellular network. Also the signal strengthand/or signal-to-noise ratio of a signal received from a base stationmay be used to estimate the distance of a device from a base station oraccess point. Furthermore, using the transmission signal from three ormore base stations or access points and their corresponding ID, a devicemay determine the location using triangulation measurements.

Such triangular methods could also be applied relative to knownlandmarks. The location could then be estimated by detecting the objectsfrom a camera image using additional compass information. Alternatively,the known landmarks may for instance emit a beacon signal (e.g. an audiosignal, a visual signal or an audiovisual signal). For example, inmarine navigation, the visual signal may have different colours orsignal codes toward different directions. The Direction of Arrival (DoA)of the emitted audio signal of the known beacon, on the other hand,could be determined using microphone array methods.

FIG. 4 is a schematic illustration of an apparatus 130 comprised in aserver of a system for location and/or orientation estimation accordingto an embodiment of the present invention. This apparatus 130 may becomprised in server 13 in system 1 of FIG. 1 a.

Apparatus 130 comprises a processor 135 that executes program codestored in program memory 131. Main memory 132 is used by processor 135for instance as working memory. Processor 135 interacts with an optionaluser interface 134 that allows interaction between a user and server 13.Finally, processor 135 interacts with a communication interface 133 thatallows communicating with other devices, for instance with first-typedevice 11 and second-type device 12 of system 1 of FIG. 1 a.

The circuitry formed by the components of apparatuses 110, 120 and 130may be implemented in hardware alone, partially in hardware and insoftware, or in software only, as further described at the end of thisspecification.

FIG. 5 is a schematic illustration of a tangible storage medium 60according to an embodiment of the present invention. This tangiblestorage medium may for instance form at least a part of program memory111 of the apparatus 110 of FIG. 2, or of program memory 121 ofapparatus 120 of FIG. 3 or of program memory 131 of apparatus 130 ofFIG. 4. It may for instance be embodied as RAM or ROM memory, butequally well as a removable memory. Tangible storage medium 60 comprisesa computer program 61, which in turn comprises program code 62. Thisprogram code may for instance implement the methods of flowchart 200 ofFIG. 6, of flowchart 300 of FIG. 7, of flowchart 400 of FIG. 8 a, offlowchart 900 of FIG. 8 b, of flowchart 1000 of FIG. 8 c, or offlowchart 1100 of FIG. 8 d, which will be discussed below.

Reconsider the system 1 of FIG. 1 a, where estimation of the locationand/or orientation of the first-type device 11 is performed by server13. The method steps performed by the first-type device 11, thesecond-type device 12 and the server 13 in this set-up are illustratedby the flowcharts 200, 300 and 400 of FIGS. 6, 7 and 8 a, respectively.

Flowchart 200 (see FIG. 6) illustrates the processing at first-typedevice 11 (see FIG. 1 a).

Flowchart 200 starts with an optional step 201, in which a coarselocation and/or an orientation of the first-type device are obtained (bycoarse location unit 116 and orientation unit 117 of apparatus 110 ofFIG. 2).

In a step 202, signals from one or more signal sources are received atthe first-type device (via sensor(s) 113 of apparatus 110 of FIG. 2).

In an optional step 203, the received signal (s) are processed (byprocessor 118 of apparatus 110 of FIG. 2). This processing may forinstance comprise separating signals received from different signalsources. This processing may furthermore comprise analyzing the receivedsignals, for instance to derive parameters such as directions betweenthe first-type device and the respective signal source from which thesignal was received, or the distance between the first-type device andthe respective signal source from which the signal was received, to namebut a few non-limiting examples. Instead of these parameters, differentparameters that nevertheless allow derivation of the direction and/ordistance may be derived.

In a step 204, information related to the signal(s) received at thefirst-type device is communicated to server 13 (via communicationinterface 114 of apparatus 110 of FIG. 2). This information may forinstance be at least partially scrambled for privacy/security reasons.This information may for instance comprise the received signals or partsthereof themselves, for instance as a sequence of sampled and quantizedvalues, or parameters derived from some or all of the received signals,such as for instance directions and/or distances towards signal sources.If, in step 201, a coarse location and/or an orientation of thefirst-type device has been obtained, this information may also becommunicated to server 13 in step 204. Furthermore, if derivedparameters are communicated, it may be advantageous that these derivedparameters are furnished with information that allows associating theseparameters with the signal source(s) they pertain to and/or with thereceived signals from which they were derived.

Flowchart 300 of FIG. 7 illustrates the processing performed by thesecond-type device 12 (see FIG. 1 a).

In a step 301, the location of the second-type device is obtained (bylocation unit 126 of apparatus 120 of FIG. 3).

In an optional step 302, the orientation of the second-type device isobtained (by orientation unit 127 of apparatus 120 of FIG. 3).

In a step 303, then signals from one or more signal sources are receivedat the second-type device (via sensor(s) 123 of apparatus 120 of FIG.3).

In an optional step 304, the received signal (s) are processed (byprocessor 128 of apparatus 120 of FIG. 3). This processing may forinstance comprise separating signals received from different signalsources. This processing may furthermore comprise analyzing the receivedsignals, for instance to derive parameters such as directions betweenthe second-type device and the respective signal source from which thesignal was received, or the distance between the second-type device andthe respective signal source from which the signal was received, to namebut a few non-limiting examples. Instead of these parameters, differentparameters that nevertheless allow derivation of the direction and/ordistance may be derived. This processing may furthermore comprisedetermining location estimate for the signal sources (for instance basedon the estimated directions and/or distances and the obtained location(and the optionally obtained orientation) of the second-type device).

In a step 305, information related to the signal(s) received at thesecond-type device and related to the obtained location of thesecond-type device is communicated to server 13 (via communicationinterface 124 of apparatus 120 of FIG. 3). This information may forinstance be at least partially scrambled for privacy/security reasons.This information may for instance comprise the received signals or partsthereof themselves, for instance as a sequence of sampled and quantizedvalues, or parameters derived from some or all of the received signals,such as for instance directions and/or distances towards signal sources,and the obtained location (and the optionally obtained orientation) ofthe second-type device. Equally well, this information may comprise (orconsist of) location estimates for signal sources determined by thesecond-type device. Furthermore, if derived parameters are communicated,it may be advantageous that these derived parameters are furnished withinformation that allows associating these parameters with the signalsource(s) they pertain to and/or with the received signals from whichthey were derived.

Flowchart 400 in FIG. 8 a illustrates the processing performed by theserver 13 (see FIG. 1 a).

In step 401, information related to the signal(s) received at thefirst-type device 11 and the second-type device(s) 12 (in case thatthere are more than one second-type device) and the location(s) of thesecond-type device(s) 12 are obtained (via communication interface 133of apparatus 130 of FIG. 4). If a coarse location and/or orientation ofthe first-type device 11 have been communicated by the first-type device11, this information would also be received in step 401, as well asoptionally obtained orientation(s) of the second-type device(s) 12.

In a step 402, the location estimate(s) for the signal source(s) aredetermined at least based on the received information related to thereceived signal(s) at the second-type device(s) and on the receivedlocation(s) (and, if available, the received orientation(s)) of thesecond-type device(s). If this information has been scrambled beforecommunication, it is accordingly descrambled. If the information relatedto the received signal(s) at the second-type device(s) comprises thereceived signal(s) in basically unprocessed form, this determining maycomprise separating, with respect to each second-type device, thereceived signal(s) stemming from different signal sources, and/oranalyzing the received signal(s) to derive parameters such as thedirection(s) and/or the distance(s) between the signal source (s) andthe second-type device(s). This analysis may for instance yield positionrelationships, each between a signal source and a second-type device,and together with the location of this second-type device (and itsorientation, if available), a location estimate for the signal sourcemay then be determined. The location estimates for the same signalsource derived based on signals received at different second-typedevices may be combined to obtain a more accurate location estimate forthis signal source. Therein, location estimates pertaining to the samesignal source may be grouped to allow this combination, the groupingbeing based on an analysis (e.g. by cross-correlation) of the receivedsignals based on which these location estimates were determined. Furtherinformation such as previous location estimates for signal sources andlevel differences of signals of different signal sources received at thesame second-type device may be considered in the determining of thelocation estimates as well.

In a step 403, position relationship(s) between the signal source(s) andthe first-type device are estimated at least based on the receivedinformation related to received signal(s) at the first-type device (andthe orientation of the first-type device, if available). Similarly, ifthe information related to the received signal(s) at the first-typedevice(s) comprises the received signal(s) in basically unprocessedform, this determining may comprise separating the received signal(s)stemming from different signal sources, and/or analyzing the receivedsignal(s) to derive parameters such as the direction(s) and/or thedistance(s) between the signal source(s) and the first-type device.

In a step 404, the location and/or orientation of the first-type deviceis estimated at least based on the determined location estimate(s) forthe signal source(s) of step 402 and on the estimated positionrelationship (s) of step 403. Therein, those of the determined locationestimate(s) and the estimated position relationship(s) that pertain tothe same signal sources are considered for setting upsignal-source-specific constraints for the location and/or orientationof the first-type device, and these signal-source-specific constraintsare merged to obtain an estimate of the location and/or orientation ofthe first-type device. Therein, the determined location estimate(s) andthe estimated position relationship(s) that pertain to the same signalsources may for instance be identified based on analysis (e.g.cross-correlation) of the respective received signals based on which thelocation estimate(s) and the position relationship estimate(s) have beendetermined.

In this estimation, further information may be used, such as forinstance a coarse location of the first-type device if available, aprevious estimate of the location and/or orientation of the first-typedevice, and level differences between signals of different signalsources received at the first-type device, to name but a few examples.

Returning now to system 1′ of FIG. 1 b, the case that the locationand/or orientation of the first-type device 11′ is estimated byfirst-type device 11′ itself shall be considered. As already describedabove, the basic architecture of first-type device 11′ then may remainthe same as the architecture of first-type device 11 shown in FIG. 2.Only the tasks performed by processor 118 change, which can beaccomplished by letting this processor 118 execute the steps offlowchart 900 of FIG. 8 b. Both the architecture and the processing ofthe second-type device 12′ remain the same as that of second-type device12 (see FIG. 3 and FIG. 7), but it has to be kept in mind thatsecond-type device 12′ now communicates with first-type device 11′instead of server 13.

Flowchart 900 of FIG. 8 b illustrates the processing of first-typedevice 11′ in the system 1′ of FIG. 1 b, when the location and/ororientation of the first-type device is estimated by the first-typedevice 11′.

In a step 901, the signal(s) from the signal source(s) are received atthe first-type device 11′ (via sensors 113 of apparatus 110 of FIG. 2).

In a step 902, information related to the signal(s) received at thesecond-type device(s) 12′ and the location(s) (and orientation (s) ifavailable) of the second-type device (s) are received. This informationis communicated by the second-type device(s) in step 305 of flowchart300 of FIG. 6.

In a step 903, the location estimate(s) for the signal source(s) aredetermined at least based on the received information from thesecond-type device(s). This is accomplished as described with referenceto step 402 of flowchart 400 of FIG. 8 a.

In a step 904, the position relationship (s) between the signalsource(s) and the first-type device are estimated at least based on thesignals received at the first-type device in step 901. This isaccomplished as described with reference to step 403 of flowchart 400 ofFIG. 8 a.

Finally, in a step 905, the location and/or orientation of thefirst-type device is estimated at least based on the determined locationestimate(s) for the signal source(s) of step 903 and the estimatedposition relationship(s) of step 904, in a way as described withreference to step 404 of the flowchart 400 of FIG. 8 a.

Once more returning to system 1′ of FIG. 1 b, the case that the locationand/or orientation of the first-type device 11′ is estimated bysecond-type device 12′ shall now be considered. The basic architectureof second-type device 12′ then may remain the same as the architectureof second-type device 12 shown in FIG. 3. Only the tasks performed byprocessor 128 change, which can be accomplished by letting thisprocessor 128 execute the steps of flowchart 1000 of FIG. 8 c. Both thearchitecture and the processing of the first-type device 11′ remain thesame as that of first-type 11 (see FIG. 2 and FIG. 6), but it has to bekept in mind that first-type device 11′ now communicates withsecond-type device 12′ instead of server 13.

Flowchart 1000 of FIG. 8 c illustrates the processing of second-typedevice 12′ in the system 1′ of FIG. 1 b, when the location and/ororientation of the first-type device is estimated by the second-typedevice 12′.

In a step 1001, reception of the signal(s) from the signal source(s)takes place (via sensors 123 of apparatus 120 of FIG. 3). For instance,the second-type device may continuously (or at least continuously duringlimited time periods) record signals from signal sources.

In a step 1002, a location of the second-type device is obtained (bylocation unit 126 of apparatus 120 of FIG. 3).

In an optional step 1003, an orientation of the second-type device isobtained (by orientation unit 127 of apparatus 120 of FIG. 3).

In a step 1004, information related to the signal (s) received at thefirst-type device 11′ is received (via communication interface 124 ofapparatus 120 of FIG. 3). Such information is communicated by thefirst-type device in step 204 of flowchart 200 of FIG. 6. When receivinginformation related to the signals received at the first-type device,the second-type device may then for instance compare this information(for instance the signals received at the first-type device) with thesignals received at the second-type device in step 1001 to identifysignals that stem from the same signal source.

Step 1005 is an optional step in which information related to one ormore signals received at one or more other second-type devices and thelocation(s) (and orientations if available) of the other second-typedevice(s) are received.

In step 1006, the location estimate(s) for the signal source(s) aredetermined at least based on the signal(s) at the second-type device instep 1004, the location (and the orientation if available from step1003) of the second-type device obtained in step 1002 and theinformation from the other second-type device (s) optionally received instep 1005. The location estimate(s) are determined as described withreference to step 402 of flowchart 400 of FIG. 8 a above.

In step 1007, the position relationship(s) between the signal source(s)and the first-type device are estimated at least based on theinformation received from the first-type device in step 1001. This isaccomplished as described with reference to step 403 of flowchart 400 ofFIG. 8 a above.

Finally, in step 1008, the location and/or orientation of the first-typedevice is estimated at least based on the location estimate (s) for thesignal source(s) determined in step 1006 and the position relationship(s) estimated in step 1007. This is accomplished as described withreference to step 404 of flowchart 400 of FIG. 8 a above.

Returning now to system 1″ of FIG. 1 c, the case that the locationand/or orientation of the first-type device 11″ that comprises signalsource 10″ is estimated by first-type device 11″ itself shall beconsidered. As already described above, the basic architecture offirst-type device 11″ then may remain the same as the architecture offirst-type device 11 shown in FIG. 2. Only the tasks performed byprocessor 118 change, which can be accomplished by letting thisprocessor 118 execute the steps of flowchart 1100 of FIG. 8 d. Both thearchitecture and the processing of the second-type device 12″ remain thesame as that of second-type device 12 (see FIG. 3 and FIG. 7), but ithas to be kept in mind that second-type device 12″ now communicates withfirst-type device 11″ instead of server 13.

Flowchart 1100 of FIG. 8 d illustrates the processing of first-typedevice 11″ in the system 1″ of FIG. 1 c, when the location and/ororientation of the first-type device is estimated by the first-typedevice 11′, which comprises the signal source 10″.

In a step 1101, a signal is transmitted form a signal source that isinternal (i.e. comprised in the first-type device), like signal source10″ of first-type device 11″ in FIG. 1 c.

In an optional step 1102, one or more respective signal(s) from one ormore signal source(s) that are external with respect to (i.e. notcomprised in) the first-type device 11″ are received at the first-typedevice 11″ (via sensors 113 of apparatus 110 of FIG. 2). Step 1102 mayfor instance only be performed if apparatus 110 of FIG. 2 is equippedwith any sensors 113 at all).

In a step 1103, information related to the signal (s) received at thesecond-type device(s) 12″ from the internal signal source and optionallyfrom one or more external signal source(s) and the location(s) (andorientation(s) if available) of the second-type device(s) are received.This information is communicated by the second-type device(s) in step305 of flowchart 300 of FIG. 6.

In a step 1104, the location estimate(s) for the internal signal sourceand the optional external signal source(s) are determined at least basedon the received information from the second-type device(s). This isaccomplished as described with reference to step 402 of flowchart 400 ofFIG. 8 a. As further information for the determination of the locationestimate for the internal signal source, the power level used for thetransmission of the signal in step 1101 may be exploited, for instanceto determine the distance between the internal signal source and thesecond-type device(s) that received signals from the internal signalsource.

In an optional step 1105, the position relationship(s) between theexternal signal source(s) and the first-type device are estimated atleast based on the signals received at the first-type device in step1102. This is accomplished as described with reference to step 403 offlowchart 400 of FIG. 8 a.

Finally, in a step 1106, the location and/or orientation of thefirst-type device is estimated at least based on the determined locationestimate(s) for the internal signal source and the optional externalsignal source(s) of step 1104, on the a priori known positionrelationship between the internal signal source and the first-typedevice and the estimated position relationship(s) between the optionalexternal signal source(s) and the first-type device of step 1105,similarly as described with reference to step 404 of the flowchart 400of FIG. 8 a.

In the following, an even more detailed description of embodiments ofthe present invention will be given with reference to FIGS. 9-20.

Therein, it is—merely as a non-limiting example—assumed that the signalsources are audio sources that emit audio signals which are detectableat receiving devices via microphones. If more than one microphone isavailable at the receiving device, a Direction of Arrival (DoA) of thereceived signal can be determined, as will be described below. It isreadily clear to a person skilled in the art that the principles ofestimating the DoA of an audio signal based on an array of microphonesalso apply to a variety of other signal types, such as for instanceelectro-magnetic signals in case that an antenna array for reception isavailable.

The basic DoA estimation is conducted using a microphone arraycomprising at least two microphones. Typically, the output of the arrayis the sum signal of all microphones. Physically turning the array anddetecting the direction that provides the highest amount of energy ofthe signal of interest is the most straightforward method to estimatethe DoA.

Steering of the array, i.e. turning the array towards the point ofinterest is typically implemented, instead of physically turning thedevice, by exploiting the sound wave interference phenomena andadjusting the microphone delay lines. For example, the two-microphonearray is aligned off the perpendicular axis of the microphones bydelaying one of the two microphone signals with respect to the other bya certain amount before summing them up. The time delay providing themaximum energy of the sum signal of interest corresponds to the DoA.

FIG. 9 illustrates the situation when the microphone array comprisingmicrophones 81 and 82 is physically turned slightly off the sound source80 (i.e. the normal of the array does not point towards sound source80). As can be seen from FIG. 9, the sound wave (illustrated by thecircle segments 84 arrives slightly delayed to the microphone 82 ascompared to microphone 81. Now, to steer the microphone array towardsthe actual sound source 80, the signal from microphone 81 needs to bedelayed, with respect to the signal from microphone 82, by a time unitcorresponding to the difference 83 in distance perpendicular to thesound source 80.

When the distance between the microphones 81 and 82, the required timedelay and the speed of sound are known, the DoA of the sound source 80with respect to the array normal may be determined using trigonometry.

Another relatively straightforward method to estimate the DoA is todetect the amplitude difference between signals captured by microphones81 and 82 and to apply corresponding panning laws.

As already discussed above, signals received at the first-type deviceand/or at the second-type device may be parameterized, for instance toreduce the amount of bandwidth required to communicate informationrelated to the received signals to the server 13 (in the system of FIG.1 a) or to either the first-type device 11′ or second-type device 12′(in the system of FIG. 1 b) or to the first-type device 11″ (in thesystem of FIG. 1 c).

In case of audio signals, an example of spatial audio parameterizationand coding is the Binaural Cue Coding (BCC), which is based on spatialcues describing the audio source locations and diffuseness in time andspace. Typically, the input signal (consisting of two or more audiochannels or sources) for a BCC encoder is first transformed intime-frequency domain using for example a Fourier transform or aQuadrature Mirror Filter (QMF) filterbank decomposition. The audio sceneis analysed in the transform domain, and the correspondingparameterization is determined and provided, e.g. transmitted, to areceiver or stored for later consumption or delivery.

Typically a BCC analysis comprises computation of Inter-channel LevelDifference (ILD) and Inter-channel Time Difference (ITD) parametersestimated within one or more transform domain time-frequency slots, i.e.in one or more frequency subbands of one or more input frames. Inaddition, the Inter-Channel Coherence (ICC) between one or more channelpairs may be determined to estimate the diffuseness of sound sources.

FIG. 10 shows an example of ILD and ITD estimation for a multi-channelaudio content (with C audio sources). The ILD and ITD parameters maytypically be determined for each channel pair. The ICC, on the otherhand, may typically be determined individually for each channel. In caseof a binaural audio signal consisting of two channels, the BCC cues maybe determined for decomposed left and right channels.

In the following, some details of the BCC approach are illustrated usingan example with two input channels and a single downmix signal. However,the representation can be generalized to cover input signal with morethan two channels.

The ILD ΔL_(n) for each frequency subband n can be estimated in thelogarithmic domain as:

$\begin{matrix}{{\Delta\; L_{n}} = {10{\log_{10}\left( \frac{s_{n}^{L^{T}}s_{n}^{L}}{s_{n}^{R^{T}}s_{n}^{R}} \right)}}} & (1)\end{matrix}$where S_(n) ^(L) and S_(n) ^(R) are the time-domain left and rightchannel signals in frequency subband n, respectively.

The ITD τ_(n), i.e. the delay, or time difference, between the left andright channels for frequency subband n, can be determined as follows:τ_(n)=arg max_(d){Φ_(n)(k,d)}  (2)where Φ_(n)(k,d) is the normalised correlation in frequency subband n:

$\begin{matrix}{{\Phi_{n}\left( {k,d} \right)} = \frac{{s_{n}^{L}\left( {k - d_{1}} \right)}{s_{n}^{R}\left( {k - d_{2}} \right)}}{\sqrt{\left( {{s_{n}^{L}\left( {k - d_{1}} \right)}{s_{n}^{L}\left( {k - d_{1}} \right)}} \right)\left( {{s_{n}^{R}\left( {k - d_{2}} \right)}{s_{n}^{R}\left( {k - d_{2}} \right)}} \right)}}} & (3)\end{matrix}$whered ₁=max{0,−d}d₂=max{0, d}  (4)

The normalised correlation of Equation (3) is actually the ICCparameter. It may typically be utilised for capturing the ambientcomponents that are decorrelated with the “dry” sound componentsrepresented by magnitude (i.e. level) and phase (time) differenceparameters of Equations (1) and (2).

Alternatively, BCC coefficients may be determined in the DFT domain.Using for example the windowed Short Time Fourier Transform (SIFT), thefrequency subband signals above are converted to groups of transformcoefficients. S_(n) ^(L) and S_(n) ^(R) are the spectral coefficientvectors of the left and right (binaural) signal for frequency subband nof the given analysis frame, respectively. The transform-domain ILD maybe determined according to Equation (5):

$\begin{matrix}{{{\Delta\; L_{n}} = {10{\log_{10}\left( \frac{S_{n}^{L^{*}}S_{n}^{L}}{S_{n}^{R^{*}}S_{n}^{R}} \right)}}},} & (5)\end{matrix}$

where * denotes the complex conjugate.

However, the time difference (ITD) may be more convenient to handle asthe Inter-Channel Phase Difference (ICPD) according to Equation (6):φ_(n)=∠(S _(n) ^(L) *S _(n) ^(R))  (6)

The inter-channel coherence may be computed in the frequency domainusing a computation quite similar to the one used in the time-domaincalculation of Equation (3) using Equation (7):

$\begin{matrix}{\Phi_{n} = \frac{S_{n}^{L^{*}}S_{n}^{R}}{\sqrt{\left( {S_{n}^{L^{*}}S_{n}^{L}} \right)\left( {S_{n}^{R^{*}}S_{n}^{R}} \right)}}} & (7)\end{matrix}$

An alternative BCC determination in the DFT domain (e.g. based onEquations (5) to (7)) may require less computation, when the time-domainITD estimation using correlation estimation is replaced with ICPD phaseestimation of DFT-domain spectral coefficients.

The level and time/phase difference cues represent the dry surroundsound components, i.e. they can be considered to model the sound sourcelocations in space.

Basically, ILD and ITD cues represent surround sound panningcoefficients. The coherence cue, on the other hand, is supposed to coverthe relation between coherent and decorrelated sounds. The level of latereverberation of the sound sources e.g. due to the room effect, and theambient sound distributed between the input channels may have asignificant contribution to the perceived spatial audio sensation.Therefore, a proper estimation and synthesis of IC cue is a matter ofimportance especially in coding and reconstruction of binaural signals.

The output of a multi-channel or binaural encoder may for instancecomprise the interchannel level difference (ILD) representing stereopanning coefficients, the interchannel phase difference (ICPD) i.e theinterchannel time difference (ITD), the interchannel correlation (ICC)and the downmix audio signal or signals.

Now, when the BCC parameterisation is available, the DoA estimation canbe conducted for each frequency subband n by first converting therespective time difference cue τ_(n) into a reference DoA cue φ_(n) bysolving the Equation (8):τ_(n)=(|x|sin(φ_(n)))/c,  (8)where |x| is the distance between the microphones and c is the speed ofsound.

Alternatively or additionally, the inter-channel level cue may be usedas basis for determining the DoA cue φ_(n). The DoA cue φ_(n) may bedetermined using for example the panning Equation (9):

$\begin{matrix}{{{\sin\;\phi_{n}} = \frac{l_{1} - l_{2}}{l_{1} + l_{2}}},} & (9)\end{matrix}$

where l_(i)=s_(n) ^(i) ^(T) s_(n) ^(i) of channel i, i.e. the energy ofchannel i in frequency subband n.

Consequently, step 203 of flowchart 200 of FIG. 6 or step 304 offlowchart 300 of FIG. 7 could analyze the receive signal (s) to deriveone or more of the above-described cues, and then this cues could becommunicated to allow a receiver to derive a DoA from these one or morecues.

In the following, an embodiment of the present invention will beconsidered that uses audio sources as signal sources and provides amethod to collaboratively estimate the accurate location and/ororientation of a (fixed or mobile) first-type device in a multi-devicesystem. This approach may for instance be used in a multi-viewapplication.

In this embodiment, a (fixed or mobile) second-type device equipped withaccurate location and direction estimation, such as GPS, knowledge ofcell ID, WLAN access points and compass, sends location and orientationinformation and at least a segment of a captured audio signal (i.e. atleast a segment of an audio signal received from one or more audiosources with a multi-microphone (array) system) to a location service,which may for instance be a dedicated service linked to a multi-viewaudiovisual media capture application or service provision or linked tomulti-user event casting. Alternatively, the second-type device mayanalyse the segment of the captured audio signal and provide the spatialaudio cues and context information to the location service. In bothcases, the intention is allow extraction of the directions or locationsof individual audio source in the captured audio scene. In addition,when the segment of the captured audio signal is provided to thelocation service, the segment of the captured audio signal may bescrambled in such a manner that it cannot be reconstructed in humanunderstandable form for privacy preservation, but that the audio scenecontext can still be reconstructed.

In this embodiment, a first-type device not equipped with accuratelocation and/or orientation estimation capability may perform roughlocation estimation using the cell ID or WLAN AP knowledge.Alternatively, no location info may be available to this first-typedevice at all. To enable location and/or orientation estimation, thefirst-type device sends at least a segment of an audio signal of shortduration to the location service. The recorded audio content comprisestwo or more channels to enable accurate spatial audio scene analysis.Alternatively, the first-type device may send determined spatial audioscene cues and context. The first-type device may also already providethe estimated audio source locations and spatial audio cues for furtheranalysis in the server. The location service then estimates the locationand/or orientation of the first-type device based on the data feeds ofthe first-type device and the second-type device(s) comparing thesensory information (cell ID, WLAN AP knowledge etc.) and spatial audiocues.

In this embodiment, alternatively a service application in thesecond-type device may parameterize the audio scene and send it to thefirst-type device without using any centralised server or service. Thefirst-type device may then perform the cell-ID- and/or WLAN-AP-basedlocalization assisted with the audio-based localisation. Alternatively,the second-type device may be a fixed system specifically designed toanalyse the audiovisual content within a certain location. In this casethe explicit location and orientation of the second-type device may beknown a priori.

This collaborative estimation of the location and/or orientation of thefirst-type device is improved when more information about theaudiovisual image is gained. Additional sensory information, such asorientation, may improve the accuracy significantly.

Setting out from this embodiment of the present invention, examples forthe estimation of the location and/or orientation of the first-typedevice based on an according analysis of signals received from one ormore audio sources at the first-type device and at a second-type devicewill now be considered. Therein, it should be noted that these examplesequally well apply to scenarios where other types of signal sources,such as for instance electromagnetic signal sources, are used.

FIG. 11 presents an example scenario in which the second-type device 12has gained an accurate location and orientation information using GPSand compass information. In addition, the DoA of the audio source 10(which is—as an example—considered to be a human speaker here) isdetected by both the second-type device 12 and the first-type device 11.Both devices 11 and 12 provide their information to the location serverin which the audio analysis is done based on the given location andorientation information of the second-type device 12 and the audiosource DoA detected in both devices 11 and 12.

FIG. 12 is a schematic illustration of an example for estimating thelocation of the first-type device of FIG. 11 according to an embodimentof the present invention under the assumption that the orientation ofthe first-type device 11 is known. Based on the location 20 of thesecond-type device and on the estimated direction 21 between thesecond-type device and the audio source (and an estimated or assumeddistance between the second-type device and the audio source), alocation estimate 22 for the audio source is determined. If the audiosource would be comprised in the first-type device, the positionrelationship between the first-type device and the audio source would bea priori known and would be that both positions (at least substantially)collocate, and then location estimate 22 for the audio source would alsorepresent the location estimate for the first-type device.

In FIG. 12, the location estimate 22 is depicted as an area rather thana point to indicate an inherent estimation uncertainty. This areareflects an uncertainty of the direction estimate and an uncertainty ofthe distance estimate. The uncertainty (e.g. width) of the directionestimate is proportional to the inter channel coherence of the signal inthe given direction (The higher the correlation, the smaller theuncertainty of the direction estimate. Low correlation, which forinstance occurs for far away sources, thus means a “blurred” locationestimate (with reverberation etc)).

Since the uncertainty of the distance estimate is considered to belarger than the uncertainty of the direction estimate, the locationestimate 22 is schematically illustrated as an ellipse in FIG. 12Alternatively, for instance also a cone-shaped location estimate or alocation estimate with a different shape could be used in FIG. 12.

The relative direction between this audio source and the first-typedevice has also been estimated by the first-type device as a positionrelationship. Therein, since the orientation of the first-type device isknown (it is presently the same as that of the second-type device), thisrelative direction can be transformed into the absolute direction 23that uses the same coordinate system as the direction 21 and thelocation estimate 22. By combining the location estimate 22 and theposition relationship represented by direction 23, a constraint 24, i.e.a location estimate 24 of the first-type device, can be obtained. Sincein the present example, initially not even a coarse location estimatefor the first-type device is available, there is no anchor point for thedirection 23, so that this direction 23 has to be represented by aplurality of parallel directions 23 (two of which are shown in FIG. 12).However, since the extension of the location estimate 22 for the audiosource is limited, the location estimate 24 can be obtained by fittingtwo directions 23 at the outer ends of location estimate 22 and assumingthe location estimate 24 to be located between these two directions 23,wherein in the present example, a circular location estimate 24 isassumed.

As can be seen from the example of FIG. 12, when only detecting thedirection of a single audio source within the audio image, only a ratherrough location estimate 24 of the first-type device is obtained, even ifboth the first-type device and the second-type device know their(relative) orientation.

It should be noted that, if the orientation of the first-type devicewould not be known, one audio source does not provide any informationabout the location of the first-type device, since, depending on theactual orientation of the first-type device, the first-type device mayin principle locate anywhere relative to the second-type device.

Furthermore, even when the orientation of the first-type device is known(as in the example of FIG. 12), the fact that the DoA-estimation doesnot provide any information about the actual distance between the audiosource and the microphone array may cause some uncertainty to thecollaborative location estimation of the first-type device. Thus,although the DoA is known accurately, the audio source can locate on anyposition along the DoA line 21 seen by the second-type device as shownin FIG. 12.

However, to mitigate this effect, the maximum distance to the audiosource may be determined based on the microphone array sensitivity(since this sensitivity may be representative of the maximum possibledistance between an audio source and the microphone array, becausefarther located audio sources may no longer be received). Furthermore,audio sources far away from the device do not necessarily produce anaccurate (and thus applicable) DoA estimate anyway.

It should be noted that the collaborative estimation of the position ofa first-type device according to the present invention also works if thesecond-type device is not capable of estimating a direction towards asignal source, for instance if it only has a single sensor/microphone.In this case, also knowledge of the orientation of the second-typedevice is not necessary. In this case, based on the known location ofthe second-type device, and a determined or assumed distance towards thesound source, a circular location estimate for the sound source can beset up centred at the location of the second-type base station, and thiscircular location estimate can then be used together with the estimatedposition relationship between the first-type device and this soundsource to estimate the position of the first-type device. Therein, theposition relationship between the first-type device and this soundsource may also only be based on a determined or assumed distancebetween the first-type device and the sound source, or on a direction(and optionally a distance) estimate together with knowledge on theorientation of the first-type device. The resulting estimates of thelocation of the first-type device may be somewhat coarse, but maynevertheless be useful depending on the application field.

It should also be noted that exploitation of signals received from thesame sound source at at least two second-type devices with knownlocations may allow dispensing with the requirement that the orientationof the second-type devices has to be known. For instance, if thelocation of each second-type device, the respective directions towardsthe sound source and the respective distances are known, a locationestimate for the sound source can be determined without requiring theorientation of the second-type devices to be known. The accuracy of thelocation estimate for the signal source further improves if more thanone signal source are received by the second-type devices. This will bediscussed further below, where it is however assumed that theorientation of the second-type device(s) is known.

Returning to the examples of FIGS. 11 and 12 above, remember that theDoA estimation in these examples was only detecting a single audiosource. However, the same parameterisation could be used for multipleaudio sources (and also other signal sources) as well. Therein, theseaudio sources or signal sources may at least partially be comprised inthe first-type device itself.

Statistical analysis of the cues could reveal that the audio scene maycontain more than one source. For example, the spatial audio cues acrossfrequency bands could be clustered in an arbitrary number of subsetsusing for example Gaussian Mixture Models (GMM).

The achieved DoA cues can be classified within M Gaussian mixtures bydetermining the Probability Density Function (pdf) function of the DoAdata for each sub band

$\begin{matrix}{{{f_{X|\theta}\left( \phi \middle| \theta \right)} = {\sum\limits_{i = 1}^{M}{\rho_{i}{f_{X|\theta_{i}}\left( \phi \middle| \theta_{i} \right)}}}},} & (10)\end{matrix}$where ρ_(i) is the component weight and components are Gaussian

$\begin{matrix}{{{f_{X|\theta_{i}}\left( \phi \middle| \theta_{i} \right)} = {\frac{1}{\sigma_{i}\sqrt{2\pi}}{\mathbb{e}}^{{{- {({\phi - \mu_{i}})}^{2}}/2}\sigma_{i}^{2}}}},} & (11)\end{matrix}$with mean μ_(i), variance σ_(i) ² and DoA cue φ. θ stands for the inputparameters of the pdf function, i.e. θ contains the mean and variance.

For example, an Expectation-Maximisation (EM) algorithm could be usedfor estimation of the component weight, mean and variance parameters foreach mixture in iterative manner using the achieved data set. For thisparticular case, the main interest lies in the mean parameter for eachGaussian mixture since it gives the estimate of the DoA of a pluralityof sound sources. Since the number of mixtures is most likely greaterthan the actual number of sound sources within the sound image, it maybe beneficial to concentrate on the parameters having greatest componentweight and lowest variance since they can be considered to indicatestrong point-like sound sources. Mixtures having mean values close toeach other could also be combined, for instance based on the assumptionthat they do represent the same source.

Furthermore, the parameterisation could contain classificationinformation about the content. For the location estimation purpose, thesound sources in the given directions should be identified (for instancebased on the received signals stemming from these sound sources). Bothdevices, the first-type device and the second-type device, shoulddistinguish the sound sources (or at least the received signals) andknow which source is in which direction. In addition to spectraldifferences of the sources, temporal classification could also be usedto distinguish sources with different DoA.

The estimation accuracy of the location and the orientation may beimproved significantly when more than one audio source are detected.When clustering the determined spatial audio image cues and thusdetecting two or more separate audio sources near the device, thelocation/orientation estimation can be made more accurate.

However, the accuracy may depend on the detected audio image. If theimage is not “rich” enough, that is, if the audio sources are notpopulating the image well enough, the location estimation may remaininaccurate.

FIG. 13 presents an example case where two separate audio sources 10-1and 10-2 are identified in the audio image. Using the information aboutthe DoAs of several audio sources limits the space in which thefirst-type device 11 can locate.

The corresponding improved location estimation is illustrated in FIG.14. Therein, it is once again assumed that the orientation of thefirst-type device is known via sensory information. If the first-typedevice does not have a priori knowledge about the orientation relativeto the second-type device, the location estimation may be less accurate.

In FIG. 14, setting out from the location 30 of the second-type device,the orientation of the second-type device and the DoAs 31 towards thefirst audio source (audio source 10-1 in FIG. 13) and 32 towards thesecond audio source (audio source 10-2 in FIG. 13), the locationestimates 33 for the first audio source and 34 for the second audiosource are determined. With respect to both audio sources 10-1 and 10-2,also the first-type device has determined relative DoAs, which can betransformed into absolute directions 35 towards the first audio source10-1 and 36 towards the second audio source 10-2, since the orientationof the first-type device is assumed to be known. Once again, locationestimate 33 and the position relationship represented by direction 35are considered to form a first audio-source-specific constraint, andlocation estimate 34 and the position relationship represented bydirection 36 are considered to form a second audio-source-specificconstraint, and by combining both constraints, the location estimate 37of the first-type device is obtained, which is vastly improved withrespect to the estimate 24 of FIG. 12 that was based on a single audiosource 10 (see FIG. 11).

Thus based on audio source classification, i.e. when the content incertain directions can be distinguished, the possible location of thefirst-type device is significantly limited compared to the situation ofhaving only one source. This also holds for the possible orientation ofthe first-type device (in case it is not known), as will be shown below.

FIG. 15 presents an alternative example for estimating thelocation/orientation of the first-type device 11 of FIG. 13, when theorientation of the first-type device is not known. This estimating,which can be considered to estimate both the location and orientation ofthe first-type device, or only one of them, may come to a wrong result,as can be seen by comparing the location estimate 47 obtained in FIG. 15with the location of the first-type device in FIG. 13.

It is noted that the location 40 and orientation of the second-typedevice and the estimated locations 42 and 44 of the sound sources (basedon the DoAs 41 and 43 estimated by the second-type device, respectively)are identical to their counterparts in FIG. 14.

Obviously, in FIG. 15, the location/orientation of the first-type devicehas been estimated in a valid manner (by combining the estimated DoA 46towards the first audio source 10-1 with the location estimate 44 ofthis first audio source and by combining the estimated DoA 45 towardsthe second audio source with the location estimate 42 of this secondaudio source), but the resulting location estimate 47 is neverthelesswrong.

It is thus apparent that, when using only the DoAs and sourceclassification (without knowledge on the orientation of the first-typedevice), the algorithm may fail. However, it is also apparent that thepossible location/orientation estimates are significantly limitedcompared to the case where only a single audio source is considered(compare the extension of location estimate 24 of FIG. 12 and theextension of location estimate 47 of FIG. 15). For example, since theleft and right hand sources, 10-2 and 10-1, respectively aredistinguished, the first-type device cannot be facing the second-typedevice.

From the above, it is noted that the location/orientation estimation ofthe first-type device may be improved every time new audio sources aredetected and identified.

Further collaboration could be utilised by for example by detecting therelative loudness of identified audio sources.

For instance, from the point of view of the second-type device 12 inFIG. 13, the left hand side audio source 10-2 is louder than the righthand side audio source 10-1. For the first-type device 11, the situationis reverse. Hence, by combining this information with the clustering ofspatial audio cues, the server or first-type/second-type device can makean estimate about the relative distance of the audio source to themicrophone array. Based on this analysis, the left hand side audiosource 10-2 is considered to be closer to the second-type device 12 thanthe right hand side audio source 10-1.

FIG. 16 presents the according estimation result under the assumptionthat the orientation of the first-type device is known. It is easy tosee that the location estimates 52 and 54 (which are based on theestimated DoAs 51 and 53 and the location 50 of the second-type device)of the audio sources are now very much limited to their counterparts 33and 34 in FIG. 14. Now that the left hand side source 10-2 (see FIG. 13)is determined to be closer to the second-type device 12 the locationestimate 56 of the first-type device 11 (with estimated DoAs 55 and 56towards the audio sources) is improved again. As a result, theestimation of the first-type device with orientation knowledge is fairlyaccurate.

And, even when the orientation of the first-type device is not known apriori, the possible locations of the first-type device with the giveninformation about the DoA, source classification and limited relativedistances is significantly improved. In this example, the first-typedevice can only locate on the right side of the second-type device, andthe possible orientation is also limited.

When a third audio source (or even more audio sources) is detected inthe audio image and the source classification as well as relativedistances can be estimated within reasonable accuracy, the location andorientation estimation of the first-type device is even more accuratethan in FIG. 16.

In the following, with respect to the flowcharts of FIGS. 17-19, anembodiment of the process for collaboratively estimating a locationand/or an orientation of a first-type device will again be explained inthe context of a system that uses audio sources as signal sources. It isassumed that the second-type device (s) use microphone arrays with atleast two microphones to receive respective signals from one or moreaudio sources and are thus capable of determining DoAs towards theseaudio sources. It is further assumed that the first-type device usesmicrophone arrays with at least two microphones to receive respectivesignals from one or more external audio sources and/or comprises atleast one internal audio source for emitting an audio signal.Furthermore, it is assumed that the second-type devices are capable ofdetermining their accurate location and orientation, and that both thefirst-type and second-type device are capable of separating multipleaudio sources. The flowcharts of FIGS. 17-19 present this embodiment inclosed form and are not necessarily device-specific, i.e. it may be thecase that the steps of each of the flowcharts of FIGS. 17-19 areexecuted by different devices. Furthermore, it is readily clear that theprinciples of the estimation of the location and/or orientation of thefirst-type device described below are equally well applicable toscenarios where signal sources other than audio sources are used (forinstance, electromagnetic signal sources that emit electromagneticsignals that are received with antennas).

As a start, flowchart 500 of FIG. 17 divides the estimation of thelocation and/or orientation of the first-type device according to anexample embodiment of the present invention into two categories.

First, the respective locations of the one or more sound sources (whichmay be external and/or internal with respect to the first-type device)are estimated in step 501 based on the information extracted by one ormore second-type devices.

Second, the location and/or orientation of the first-type device isestimated in step 502 based on the location estimate(s) for the soundsource(s) derived in step 501. Steps 501 and 502 themselves contain manyprocessing steps and are described in more detail with reference toFIGS. 18 and 19, respectively.

FIG. 18 presents the processing steps of flowchart 600 as an example fordetermining the location estimate (s) of the sound source(s) accordingto step 501 of flowchart 500 of FIG. 17. It is noted that the order ofsome individual steps may be altered without affecting the processingresult. For example, step 601 may be alternatively placed between steps605 and 606.

In step 601, the (absolute) location and the orientation of asecond-type device is obtained. The location may for instance be gottenfrom GPS or through any other positioning means. The orientation may begotten from a compass in the second-type device or through any otherorientation detection means.

In step 602, an audio signal is captured through the microphones in thesecond-type device. Audio capturing may be consistently on, may happenperiodically or according to any predetermined pattern, or may takeplace on request. The audio signals may stem from audio sources that areexternal with respect to the first-type devices and/or from audiosignals that are internal with respect to the first-type device.

In step 603, the sound source(s) are detected from the captured audiosignal. The sound source detection may for instance use the GaussianMixture Models (GMM) method presented earlier or any other suitablemethod.

In step 604, the direction(s) and distance(s) of the sound source(s)from the second-type device are estimated. Some methods for thedirection and distance estimation have been presented above.

In step 605, the individual sound source(s) or spatial cues are matchedwith those captured with other second-type devices, if any. The recordedsound source (s) or spatial cues are provided, for example streamed, toa server or to other second-type devices. A cross-correlation estimate,for instance similar to Equation (2) and Equation (3), of the receivedsound signals or spatial cue vectors with the locally stored contentwill determine the matched source(s). The estimated buffering delay ineach device could be taken into account when determining thecorresponding audio events to assist computation of thecross-correlation estimate. In other words, the processing in step 605results into pair-wise identification which sound source in the currentaudio signal of the current second-type device is likely to be the sameas a particular sound source detected from other second-type devices.

In step 606, the location of the sound source(s) is estimated based onthe estimated direction(s) and distance(s) from the second-type deviceand the location estimate (s) obtained from processing the informationfrom other second-type devices, if any. If no information of othersecond-type devices has been processed, the location estimate(s) aresimply derived from the location and orientation of the second-typedevice and the direction and distance estimation(s) to the soundsource(s). If an earlier location estimate for a sound source has beenderived, then it is updated e.g. by taking the cross-section of theearlier location estimate and the new estimate derived from the locationand orientation of the second-type device and the direction and distanceestimation to the sound source.

In step 607, it is checked whether there are more second-type devices inabout the same area. If there is, steps 601 to 606 are repeated for thenext second-type device. Otherwise, the location estimates for the soundsources are not refined further.

FIG. 19 presents processing steps of flowchart 700 as an example forestimating the location of the first-type device based on theinformation and location estimate(s) for the sound source (s) accordingto step 502 of flowchart 500 of FIG. 17. It is noted that the order ofsome individual steps may be altered without affecting the processingresult. For example, step 705 could be placed in other locations of theprocessing chain, such as preceding step 701.

In step 701, respective audio signals from one or more external audiosources are captured through the microphones in the first-type device,and/or an audio signal is emitted by an internal audio source comprisedin the first-type device. Audio capturing may be consistently on, mayhappen periodically or according to any predetermined pattern, or maytake place on request. If both capturing of one or more external audiosources and emission of an audio signal by an internal audio source isperformed, these actions may take place simultaneously. Alternatively,capturing of one or more external audio sources and emission of an audiosignal by an internal audio source may for instance be separated in thetime domain to avoid mutual interference; for instance, first theemission of the audio signal by the internal audio source may beperformed, for instance for a pre-defined time, and then the capturingof the external audio source(s) may be performed.

In steps 702, 703 and 704, the one or more individual external soundsources are detected from the captured audio signal (if audio capturehas taken place in step 701), the respective directions and distances ofthe one or more external sound sources are estimated (if audio capturehas taken place in step 701), and the external and/or internal soundssources are matched to those captured by the second-type device(s).Steps 702, 703 and 704 include similar processing to steps 603, 604 and605 of flowchart 600 of FIG. 18, respectively, and are therefore notdescribed here in detail.

In step 705, a rough location and orientation of the first-type deviceis obtained, if possible. A rough location may for instance be obtainedthrough the cell-ID of the cellular network in use, for example. A roughorientation may be obtained through sensors available in the device, forexample. The rough location and orientation are used as an initialconstraint for audio-based derivation of location estimates. It is notmandatory that either or both of location and orientation estimate arederived. However, initial rough estimates can make the audio basedlocation estimation more accurate as described earlier.

In step 706, the current location and orientation estimates of thefirst-type device are refined based on the estimated direction (s) anddistance (s) towards the sound source(s). The location estimate(s) forthe sound source(s) have been determined in step 501 of flowchart 500 ofFIG. 17. Given the estimated direction(s) and distance(s) between theexternal sound source(s) and the first-type device (i.e. estimatedposition relationship(s)) as well as the rough orientation of thefirst-type device, if available, and/or the a priori known positionrelationship between the internal sound source and the first-typedevice, a candidate area of the location of the first-type device can bederived.

The current location estimate for the first-type device can be formed asa cross-section of the candidate area and an earlier location estimate.

The current orientation estimate of the first-type device can be formedby excluding those orientations that are not consistent with the earlierorientation estimate(s), the current location estimate, the detectedsound source directions, and the estimated locations of the soundsources.

It is noted that both the location estimate and the orientation estimatemay typically be inaccurate and may be expressed as an area and a range,respectively, and may also be associated with probability distributionfunction across the area and the range, respectively.

In step 707, it is checked if there are more (external) sound sourcesavailable. If there is, then the step 706 is repeated for the next soundsource. If there are no more sound sources, then the location and/ororientation estimates of the first-type device cannot be refinedanymore.

Finally, with respect to FIG. 20, an example of a protocol that controlsthe interaction between the first-type device and the second-type device(and the server, if any) for collaborative location and/or orientationestimation.

The collaborative location estimation is most likely initiated by afirst-type device not having accurate means for position sensing.

In a step 801, the first-type device sends (e.g. broadcasts) acollaborative location estimation request to a server or to second-typedevices available in the neighborhood, e.g. to those having the samecell ID.

In case the server is requested for location assistance, the servermakes an enquiry for available second-type devices. Alternatively, theserver may have fixed devices within the given location (step 802).

When accepting the request, the second-type device(s) start recordingthe audio signal and analysing the location of the sound sources (step803).

At the same time, the first-type device gets a respectiveacknowledgement from the accepting second-type device(s) (step 804)directly or via a server. In response to the acknowledgement(s), thefirst-type device triggers its internal audio source to emit a signal(if such an internal audio source is available) and/or starts captureand analysis of signals from one or more external audio sources (if asensor for receiving such signals from external audio sources isavailable).

For instance after a predetermined time, (or also after beingtriggered), the second-type device(s) provide the recorded audio files,the analysis results and the accurate location and orientation cueseither directly to the first-type device or via the server (step 805),after which the first-type device conducts the location analysisdescribed above (step 806). Therein, if the first-type device emitted anaudio signal in step 804 itself, i.e. if the first-type device comprisesan audio source, this information is considered in the analysis of step806.

The predetermined recording time could be extended, or an additionalrecording and analysis session could be performed, when the achievedcontent was not rich enough, i.e. it did not contain sufficient numberof sound sources (step 807).

The delivery, e.g. streaming, of audio and location information couldalso be continuous until the location of the first-type device isestimated with sufficient accuracy, or the action is cancelled. When theaudio content from the one or more second-type device does not match thecontent recorded and/or emitted by the first-type device, the conclusioncould be that the second-type device is not in the same audioenvironment with the first-type device. In that case, another requestcould be sent or broadcasted to find other second-type devices withinthe neighborhood (step 808).

In one embodiment of the invention, both the first-type device and thesecond-type device(s) will provide, e.g. stream, the captured audiocontent and the available location information to the collaborativelocation estimation server which is then conducting the locationanalysis of the first-type device.

In another embodiment of the invention, the second-type device(s) and/orserver(s) capable for the collaborative location estimation are searchedwithin an ad-hoc network or wireless sensor network the first-typedevice connected to via WLAN, Bluetooth or any other type of radio link.

It is noted that those location estimation techniques presented abovethat are not based on reception of audio signals (for instance thelocation estimation techniques based on GNSS, triangulation or cell-IDsthat are for instance used to determine the location of the second-typedevice) are given just as examples. Other location estimation techniquescan be used before the above presented collaborative (e.g. audio-based)location estimation or may assist this location estimation.

For example, it is known that the location and/or orientation of adevice can be estimated based on images or video captured with thedevice. Feature points of a captured image are then extracted andmatched with those available in the 3D model of the surroundings. Thelocation and orientation of the device can be estimated by matchingseveral feature points to the respective feature points of the 3D modelof the surroundings. Such visual localization can be used to refine thelocation and orientation estimate of the second-type device.Furthermore, such visual localization can be used to refine the initialrough estimate of the location and orientation of the first-type device.It is noted that if no 3D model of the surroundings is readily available(e.g. through a 3D map service), feature points extracted from images ofsecond-type devices and first-type devices can be matched, and a 3Dpoint cloud be generated accordingly. However, such a point cloud mayusually not be in scale, i.e. the absolute distances of the devicesrelative to each other or relative to the objects in the images may notbe derivable. The presented collaborative (e.g. audio-based) locationestimation can be used in determining the scale for visual 3D model.

As has been described above, embodiments of the present invention allowobtaining an accurate location estimate for a (first-type) devicewithout expensive and/or power-consuming GNSS functionality or otherexpensive and/or power-consuming localization sensors. This canpotentially enable location based services for low-end phones withoutGNSS. To this end, device location estimation is done collaborativelyusing the available accurate information of one device and processingthe common sensory information available for everyone. In embodiments ofthe present invention, using the common audio scene information, such asincreasing number of discrete sources and relative loudness of thesources seen by different devices, the location estimation can beiteratively improved when more information is retrieved. In embodimentsof the present invention, the collaborative location estimation can bedone in centralised service or within an ad-hoc type network among thecollaborating devices. In embodiments of the present invention, thecomputational load can be distributed to the server by transmitting anaudio clip instead of conducting the audio scene analysis within thedevices. In embodiments of the present invention, the user privacy maynot be compromised if scrambled audio data is used for the sceneanalysis in the server (i.e. the captured audio clips are transmitted inscrambled form to the server or devices). In embodiments of the presentinvention, the second-type device could be a fixed system within theplace of interests with a priori location and orientation information.

As used in this application, the term ‘circuitry’ refers to all of thefollowing:

(a) hardware-only circuit implementations (such as implementations inonly analog and/or digital circuitry) and

(b) combinations of circuits and software (and/or firmware), such as (asapplicable):

-   -   (i) to a combination of processor(s) or    -   (ii) to portions of processor(s)/software (including digital        signal processor(s)), software, and memory(ies) that work        together to cause an apparatus, such as a mobile phone or a        positioning device, to perform various functions) and        (c) to circuits, such as a microprocessor(s) or a portion of a        microprocessor(s), that require software or firmware for        operation, even if the software or firmware is not physically        present.

This definition of ‘circuitry’ applies to all uses of this term in thisapplication, including in any claims. As a further example, as used inthis application, the term “circuitry” would also cover animplementation of merely a processor (or multiple processors) or portionof a processor and its (or their) accompanying software and/or firmware.The term “circuitry” would also cover, for example and if applicable tothe particular claim element, a baseband integrated circuit orapplications processor integrated circuit for a mobile phone or apositioning device.

The invention has been described above by means of embodiments, whichshall be understood to be non-limiting examples. In particular, itshould be noted that there are alternative ways and variations which areobvious to a skilled person in the art and can be implemented withoutdeviating from the scope and spirit of the appended claims. It shouldalso be understood that the sequence of method steps in the flowchartspresented above is not mandatory, also alternative sequences may bepossible.

The invention claimed is:
 1. A method for estimating at least one of alocation or an orientation of a first-type device, the methodcomprising: receiving respective location estimates for one or moresignal sources, wherein at least one location estimate for a signalsource of said one or more signal sources is determined at least basedon respective signals from said signal source received at one or moresecond-type devices and respective locations of said one or moresecond-type devices, and respective position relationships between saidone or more signal sources and said first-type device; estimating, bythe processor, at least one of the location or the orientation of saidfirst-type device at least based on the respective location estimatesfor the one or more signal sources and the respective positionrelationships between said one or more signal sources and saidfirst-type device, wherein said respective location estimates for saidone or more signal sources are determined at least based on saidrespective locations of said one or more second-type devices, respectiveorientations of said one or more second-type devices and respectiveestimates of respective directions between said signal sources and saidone or more second-type devices determined at least based on therespective signals received at said one or more second-type devices. 2.The method according to claim 1, wherein said one or more signal sourcesare sound sources that emit respective audio signals.
 3. The methodaccording to claim 1, wherein said at least one location estimate forsaid signal source is determined further based on respective estimatesof respective distances between said signal source and said one or moresecond-type devices.
 4. The method according to claim 1, wherein said atleast one location estimate for said signal source is determined furtherbased on at least one previously determined location estimate for saidsignal source.
 5. The method according to claim 1, wherein said at leastone location estimate for said signal source is determined further basedon level differences between respective signals from at least two ofsaid one or more signal sources received at a second-type device of saidone or more second-type devices.
 6. The method according to claim 1,wherein said respective signals received at said second-type devices areanalyzed to decide if they stem from the same signal source and are thusjointly useable as a basis for determining said at least one locationestimate for said signal source.
 7. The method according to claim 1,wherein at least one of said one or more signal sources is comprised insaid first-type device, and wherein a position relationship between saidat least one signal source comprised in said first-type device and saidfirst-type device is known a priori.
 8. The method according to claim 7,wherein said estimating of said at least one of said location and saidorientation of said first-type device is performed by said first-typedevice.
 9. The method according to claim 8, further comprising:receiving, at said first-type device, one of said respective locationestimates for said one or more signal sources and information based onwhich said respective location estimates for said one or more signalsources are derivable from at least one of said one or more second-typedevices and a service to which said one or more second-type devicesprovided one of said respective location estimates for said one or moresignal sources and information based on which said respective locationestimates for said one or more signal sources are derivable.
 10. Anapparatus for estimating at least one of a location or an orientation ofa first-type device, the apparatus comprising at least one processor;and at least one memory including computer program code, said at leastone memory and said computer program code configured to, with said atleast one processor, cause said apparatus at least to: receiverespective location estimates for one or more signal sources, wherein atleast one location estimate for a signal source of said one or moresignal sources is determined at least based on respective signals fromsaid signal source received at one or more second-type devices andrespective locations of said one or more second-type devices, andrespective position relationships between said one or more signalsources and said first-type device; estimate at least one of thelocation or the orientation of said first-type device at least based onthe respective location estimates for the one or more signal sources andthe respective position relationships between said one or more signalsources and said first-type device, wherein said respective locationestimates for said one or more signal sources are determined at leastbased on said respective locations of said one or more second-typedevices, respective orientations of said one or more second-type devicesand respective estimates of respective directions between said signalsources and said one or more second-type devices determined at leastbased on the respective signals received at said one or more second-typedevices.
 11. The apparatus according to claim 10, wherein said apparatusis said first-type device or a part thereof.
 12. A method comprising:receiving, at a first-type device, a signal from a signal source of oneor more signal sources to serve as an at least partial basis forestimating a position relationship between said signal source and saidfirst-type device, and communicating information related to said signalreceived at said first-type device from said first-type device to one ofa service or a second-type device, wherein an estimate of at least oneof a location or an orientation of said first-type device is derivableat least based on respective position relationships between said one ormore signal sources and said first-type device and respective locationestimates for said one or more signal sources, wherein at least onelocation estimate for a signal source of said one or more signal sourcesis determinable at least based on respective signals from said signalsource received at one or more second-type devices and respectivelocations of said one or more second-type devices, and wherein at leastone of said service or said second-type device is configured to estimatesaid at least one of said location and said orientation of saidfirst-type device based on said information related to said signalreceived at said first-type device and said respective locationestimates for said one or more signal sources.
 13. An apparatuscomprising at least one processor; and at least one memory includingcomputer program code, said at least one memory and said computerprogram code configured to, with said at least one processor, cause saidapparatus at least to receive, at a first-type device, a signal from asignal source of one or more signal sources to serve as an at leastpartial basis for estimating a position relationship between said signalsource and said first-type device, and communicate information relatedto said signal received at said first-type device from said first-typedevice to at least one of a service or a second-type device of saidsecond-type devices, wherein an estimate of at least one of a locationor an orientation of said first-type device is derivable at least basedon respective position relationships between said one or more signalsources and said first-type device and respective location estimates forsaid one or more signal sources, wherein at least one location estimatefor a signal source of said one or more signal sources is determinableat least based on respective signals from said signal source received atone or more second-type devices and respective locations of said one ormore second-type devices, and wherein at least one of said service orsaid second-type device estimates is configured to estimate said atleast one of said location or said orientation of said first-type devicebased on said information related to said signal received at saidfirst-type device and said respective location estimates for said one ormore signal sources.
 14. A method comprising: receiving, at asecond-type device of one or more second-type devices, a signal from asignal source of one or more signal sources to serve, together with atleast a location of said second-type device, as an at least partialbasis for determining a location estimate for said signal source, anddetermining, by a processor, a location estimate for said signal sourcebased at least in part on the signal from the signal source, whereinsaid location estimate for said signal source is determined at leastbased on said respective locations of said one or more second-typedevices, respective orientations of said one or more second-typedevices, and respective estimates of respective directions between saidsignal source and said one or more second-type devices determined atleast based on said respective signals received at said one or moresecond-type devices, and wherein an estimate of at least one of alocation or an orientation of a first-type device is derivable at leastbased on respective location estimates for said one or more signalsources, and respective position relationships between said one or moresignal sources and said first-type device.
 15. The method according toclaim 14, further comprising: communicating information related to saidsignal received at said second-type device from said second-type deviceto one of a service, said first-type device and another second-typedevice, wherein said one of said service, said first-type device andsaid other second-type device is configured to estimate said at leastone of said location and said orientation of said first-type devicebased on said information related to said signal received at saidsecond-type device and said respective position relationships betweensaid one or more signal sources and said first-type device.
 16. Anapparatus comprising at least one processor; and at least one memoryincluding computer program code, said at least one memory and saidcomputer program code configured to, with said at least one processor,cause said apparatus at least to receive, at a second-type device of oneor more second-type devices, a signal from a signal source of one ormore signal sources to serve, together with at least a location of saidsecond-type device, as an at least partial basis for determining alocation estimate for said signal source, determine a location estimatefor said signal source based at least in part on the signal from thesignal source, wherein said location estimate for said signal source isdetermined at least based on said respective locations of said one ormore second-type devices, respective orientations of said one or moresecond-type devices, and respective estimates of respective directionsbetween said signal source and said one or more second-type devicesdetermined at least based on said respective signals received at saidone or more second-type devices, and wherein an estimate of at least oneof a location or an orientation of a first-type device is derivable atleast based on respective location estimates for said one or more signalsources, and respective position relationships between said one or moresignal sources and said first-type device.
 17. The apparatus accordingto claim 16, wherein said at least one memory and said computer programcode are further configured to, with said at least one processor, causesaid apparatus to communicate information related to said signalreceived at said second-type device from said second-type device to oneof a service, said first-type device and another second-type device,wherein said one of said service, said first-type device and said othersecond-type device is configured to estimate said at least one of saidlocation and said orientation of said first-type device based on saidinformation related to said signal received at said second-type deviceand said respective position relationships between said one or moresignal sources and said first-type device.